chensong1995 / hybridpose Goto Github PK

Hey, I am the very last step at src/train_core.py but I am running into the error below:

-> print('Could not restore session properly, check the load_dir')
(Pdb)

Traceback (most recent call last):
File "/home/ubuntu/HybridPose/lib/utils.py", line 25, in load_session
model.load_state_dict(torch.load(os.path.join(args.load_dir, 'model.pth')))

File "/home/ubuntu/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/serialization.py", line 79, in validate_cuda_device
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Can I load the model without CUDA being enabled? I am running it on an Ubuntu machine, deep learning instance, from aws currently where I have not set up CUDA.

这个can.zip解压出来是就是can类的权值，那么cat类的预训练模型是否缺失了？

Hello

I'm going to use the "label.py' code you provided,
you used the cat/005.xyz model for the occlusion linemod dataset.
Can you tell me which model you used for linemod dataset?
Did you run the symseg code using [linemod/original/dataset/cat/cat.ply] model?

Also, I would appreciate it if you could share the "label.py" code written by linemod dataset.

Hi, I followed the instructions to do the training, however, the code terminated with the following error:

Exception has occurred: ImportError
/home/linfang/Documents/Code/HybridPose/lib/ransac_voting_gpu_layer/ransac_voting.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
  File "/home/linfang/Documents/Code/HybridPose/lib/ransac_voting_gpu_layer/ransac_voting_gpu.py", line 3, in <module>
    import lib.ransac_voting_gpu_layer.ransac_voting as ransac_voting
  File "/home/linfang/Documents/Code/HybridPose/trainers/coretrainer.py", line 10, in <module>
    from lib.ransac_voting_gpu_layer.ransac_voting_gpu import ransac_voting_layer_v3
  File "/home/linfang/Documents/Code/HybridPose/src/train_core.py", line 12, in <module>
    from trainers.coretrainer import CoreTrainer

When I was compiling the Ransac Voting Layer, there was a warning about the torch C++ extension. I tried to add #include <torch/extrension.h> into ransac_voting.cpp, but it didn't work. It seems that the error is related to the version of Pytorch, could you please share the torch version?

running build_ext
building 'ransac_voting' extension
gcc -pthread -B /home/linfang/anaconda3/envs/hybridpose/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/TH -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/THC -I/home/linfang/anaconda3/envs/hybridpose/include/python3.7m -c ./src/ransac_voting.cpp -o build/temp.linux-x86_64-3.7/./src/ransac_voting.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=ransac_voting -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from ./src/ransac_voting.cpp:1:0:
/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:11:4: warning: #warning "Including torch/torch.h for C++ extensions is deprecated. Please include torch/extension.h" [-Wcpp]
 #  warning "Including torch/torch.h for C++ extensions is deprecated. Please include torch/extension.h"
    ^~~~~~~
/usr/bin/nvcc -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/TH -I/home/linfang/anaconda3/envs/hybridpose/lib/python3.7/site-packages/torch/include/THC -I/home/linfang/anaconda3/envs/hybridpose/include/python3.7m -c ./src/ransac_voting_kernel.cu -o build/temp.linux-x86_64-3.7/./src/ransac_voting_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=ransac_voting -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
g++ -pthread -shared -B /home/linfang/anaconda3/envs/hybridpose/compiler_compat -L/home/linfang/anaconda3/envs/hybridpose/lib -Wl,-rpath=/home/linfang/anaconda3/envs/hybridpose/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/./src/ransac_voting.o build/temp.linux-x86_64-3.7/./src/ransac_voting_kernel.o -L/usr/lib -lcudart -o build/lib.linux-x86_64-3.7/ransac_voting.cpython-37m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.7/ransac_voting.cpython-37m-x86_64-linux-gnu.so -> 

Are there any pointers on the curation of a custom dataset which will match Hybrid Pose's data format?

I tried the visualization functions of coretrainer.py which are working pretty well!
However, I also want to visualize the pose which is calculated in the regression module.

My code looks like this for drawing a simple coordinate system of the estimated pose with R_pred, t_pred into the image. Unfortunately it' not working.

def visualize_pose(camera_intrinsic, R, t, image):
    image_pred = image.copy()
    K = np.matrix([[camera_intrinsic['fu'], 0, camera_intrinsic['uc']],
                    [0, camera_intrinsic['fv'], camera_intrinsic['vc']],
                    [0, 0, 1]], dtype=np.float32)
    rotV, _ = cv2.Rodrigues(R)
    points = np.float32([[10, 0, 0], [0, 10, 0], [0, 0, 10], [0, 0, 0]]).reshape(-1, 3)
    axisPoints, _ = cv2.projectPoints(points, rotV, t, K, (0, 0, 0, 0))
    image_pred = cv2.line(image_pred, tuple(axisPoints[3].ravel()), tuple(axisPoints[0].ravel()), (255,0,0), 3)
    image_pred = cv2.line(image_pred, tuple(axisPoints[3].ravel()), tuple(axisPoints[1].ravel()), (0,255,0), 3)
    image_pred = cv2.line(image_pred, tuple(axisPoints[3].ravel()), tuple(axisPoints[2].ravel()), (0,0,255), 3)
    #show img
    cv2.imshow('Image_pose',image_pred)
    cv2.waitKey(0)

Do you have a snipped for me which you used for the image (Figure 1 (e), (f)) of the paper?

This would be a nice enhancement for this repository.

Thanks!

I have been trying to compile the Pose Regressor but I'm getting errors like "error: ‘Eigen::Vector4d’ has no member named ‘head’." I have made sure to use eigen3 in the include of the Makefile ( -I /usr/include/eigen3). But I'm still getting these errors with Eigen.

I'm getting this error during training-> No such file or directory: 'data/occlusion_linemod/valid_poses/ape'
In the directory there is no such folder. Inside data/occlusion_linemod/ the only folders are poses, models and RGB-D.

H

I have to estimate the 6D Pose of regular shape objects(like construction blocks ) in the construction field. So it is for the grasping in pick and place task. Roughly the camera will be positioned around 1m of the object (the construction block) and I need to get the accurate position and orientation of the top blocks, The blocks can be on top of each other. Would this approach work on real-time (around 20-30fps) and accuracy of 2-3cm by 1m distance?

I have Cuda 10 and GPU card as well.

Thanks

Hi Chen,

First of all, thank you for sharing your great work. I am a bit confused with the Dataset set-up
in your description. You have stated that you have corrected Linemod pose labels, does that mean you used the labels different from the original Linemod dataset to evaluate the performance? I have downloaded the file in the link provided but could not find the corrected pose label(orientation and translation).

Thank you

So whenever I try to make, it shows that Eigen/Dense not found. I installed the eigen3 library. After make, i get this:

g++ -O3 --std=c++11 -shared -Ieigen -IDataStructure -IOperation -ILinearAlgebra -o wrapper.so -fPIC wrapper.cpp DataStructure/.cpp Operation/.cpp LinearAlgebra/.cpp
Operation/pose_regression_init.cpp: In member function ‘void PoseRegression::LeadingEigenSpace(Matrix12d&, const unsigned int&, const HybridPredictionContainer&, std::vector<Eigen::Matrix<double, 12, 1> >, AffineXform3d*)’:
Operation/pose_regression_init.cpp:238:42: error: no matching function for call to ‘Eigen::Matrix<double, 6, 1>::Matrix(double, double, double, double, double, double)’
Vector6d y(1.0, 0.0, 0.0, 1.0, 0.0, 1.0);
^
In file included from /usr/include/eigen3/Eigen/Core:367:0,
from /usr/include/eigen3/Eigen/Dense:1,
from DataStructure/hybrid_prediction_container.h:5,
from Operation/pose_regression.h:9,
from Operation/pose_regression_init.cpp:1:
/usr/include/eigen3/Eigen/src/Core/Matrix.h:389:14: note: candidate: template Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Eigen::RotationBase<OtherDerived, ColsAtCompileTime>&)
explicit Matrix(const RotationBase<OtherDerived,ColsAtCompileTime>& r);
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:389:14: note: template argument deduction/substitution failed:
Operation/pose_regression_init.cpp:238:42: note: mismatched types ‘const Eigen::RotationBase<OtherDerived, 1>’ and ‘double’
Vector6d y(1.0, 0.0, 0.0, 1.0, 0.0, 1.0);
^
In file included from /usr/include/eigen3/Eigen/Core:367:0,
from /usr/include/eigen3/Eigen/Dense:1,
from DataStructure/hybrid_prediction_container.h:5,
from Operation/pose_regression.h:9,
from Operation/pose_regression_init.cpp:1:
/usr/include/eigen3/Eigen/src/Core/Matrix.h:378:25: note: candidate: template Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Eigen::EigenBase&)
EIGEN_STRONG_INLINE Matrix(const EigenBase &other)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:378:25: note: template argument deduction/substitution failed:
Operation/pose_regression_init.cpp:238:42: note: mismatched types ‘const Eigen::EigenBase’ and ‘double’
Vector6d y(1.0, 0.0, 0.0, 1.0, 0.0, 1.0);
^
In file included from /usr/include/eigen3/Eigen/Core:367:0,
from /usr/include/eigen3/Eigen/Dense:1,
from DataStructure/hybrid_prediction_container.h:5,
from Operation/pose_regression.h:9,
from Operation/pose_regression_init.cpp:1:
/usr/include/eigen3/Eigen/src/Core/Matrix.h:370:25: note: candidate: Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>&) [with _Scalar = double; int _Rows = 6; int _Cols = 1; int _Options = 0; int _MaxRows = 6; int _MaxCols = 1]
EIGEN_STRONG_INLINE Matrix(const Matrix& other) : Base(other)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:370:25: note: candidate expects 1 argument, 6 provided
/usr/include/eigen3/Eigen/src/Core/Matrix.h:357:25: note: candidate: Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Scalar&, const Scalar&, const Scalar&, const Scalar&) [with _Scalar = double; int _Rows = 6; int _Cols = 1; int _Options = 0; int _MaxRows = 6; int _MaxCols = 1; Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Scalar = double]
EIGEN_STRONG_INLINE Matrix(const Scalar& x, const Scalar& y, const Scalar& z, const Scalar& w)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:357:25: note: candidate expects 4 arguments, 6 provided
/usr/include/eigen3/Eigen/src/Core/Matrix.h:347:25: note: candidate: Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Scalar&, const Scalar&, const Scalar&) [with _Scalar = double; int _Rows = 6; int _Cols = 1; int _Options = 0; int _MaxRows = 6; int _MaxCols = 1; Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Scalar = double]
EIGEN_STRONG_INLINE Matrix(const Scalar& x, const Scalar& y, const Scalar& z)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:347:25: note: candidate expects 3 arguments, 6 provided
/usr/include/eigen3/Eigen/src/Core/Matrix.h:301:25: note: candidate: template<class T0, class T1> Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const T0&, const T1&)
EIGEN_STRONG_INLINE Matrix(const T0& x, const T1& y)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:301:25: note: template argument deduction/substitution failed:
Operation/pose_regression_init.cpp:238:42: note: candidate expects 2 arguments, 6 provided
Vector6d y(1.0, 0.0, 0.0, 1.0, 0.0, 1.0);
^
In file included from /usr/include/eigen3/Eigen/Core:367:0,
from /usr/include/eigen3/Eigen/Dense:1,
from DataStructure/hybrid_prediction_container.h:5,
from Operation/pose_regression.h:9,
from Operation/pose_regression_init.cpp:1:
/usr/include/eigen3/Eigen/src/Core/Matrix.h:293:34: note: candidate: template Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const T&)
EIGEN_STRONG_INLINE explicit Matrix(const T& x)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:293:34: note: template argument deduction/substitution failed:
Operation/pose_regression_init.cpp:238:42: note: candidate expects 1 argument, 6 provided
Vector6d y(1.0, 0.0, 0.0, 1.0, 0.0, 1.0);
^
In file included from /usr/include/eigen3/Eigen/Core:367:0,
from /usr/include/eigen3/Eigen/Dense:1,
from DataStructure/hybrid_prediction_container.h:5,
from Operation/pose_regression.h:9,
from Operation/pose_regression_init.cpp:1:
/usr/include/eigen3/Eigen/src/Core/Matrix.h:273:5: note: candidate: Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>&&) [with _Scalar = double; int _Rows = 6; int _Cols = 1; int _Options = 0; int _MaxRows = 6; int _MaxCols = 1]
Matrix(Matrix&& other)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:273:5: note: candidate expects 1 argument, 6 provided
/usr/include/eigen3/Eigen/src/Core/Matrix.h:267:14: note: candidate: Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(Eigen::internal::constructor_without_unaligned_array_assert) [with _Scalar = double; int _Rows = 6; int _Cols = 1; int _Options = 0; int _MaxRows = 6; int _MaxCols = 1]
explicit Matrix(internal::constructor_without_unaligned_array_assert)
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:267:14: note: candidate expects 1 argument, 6 provided
/usr/include/eigen3/Eigen/src/Core/Matrix.h:259:25: note: candidate: Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix() [with _Scalar = double; int _Rows = 6; int _Cols = 1; int _Options = 0; int _MaxRows = 6; int _MaxCols = 1]
EIGEN_STRONG_INLINE Matrix() : Base()
^
/usr/include/eigen3/Eigen/src/Core/Matrix.h:259:25: note: candidate expects 0 arguments, 6 provided
In file included from /usr/include/eigen3/Eigen/Core:297:0,
from /usr/include/eigen3/Eigen/Dense:1,
from DataStructure/hybrid_prediction_container.h:5,
from Operation/pose_regression.h:9,
from Operation/pose_regression_init.cpp:1:
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h: In instantiation of ‘void Eigen::internal::call_assignment_no_alias(Dst&, const Src&, const Func&) [with Dst = Eigen::Matrix<std::complex, -1, -1>; Src = Eigen::Matrix<double, -1, -1>; Func = Eigen::internal::assign_op<std::complex >]’:
/usr/include/eigen3/Eigen/src/Core/PlainObjectBase.h:700:41: required from ‘Derived& Eigen::PlainObjectBase::_set_noalias(const Eigen::DenseBase&) [with OtherDerived = Eigen::Matrix<double, -1, -1>; Derived = Eigen::Matrix<std::complex, -1, -1>]’
/usr/include/eigen3/Eigen/src/Core/PlainObjectBase.h:510:19: required from ‘Eigen::PlainObjectBase::PlainObjectBase(const Eigen::DenseBase&) [with OtherDerived = Eigen::Matrix<double, -1, -1>; Derived = Eigen::Matrix<std::complex, -1, -1>]’
/usr/include/eigen3/Eigen/src/Core/Matrix.h:379:29: required from ‘Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Eigen::EigenBase&) [with OtherDerived = Eigen::Matrix<double, -1, -1>; _Scalar = std::complex; int _Rows = -1; int _Cols = -1; int _Options = 0; int _MaxRows = -1; int _MaxCols = -1]’
Operation/pose_regression_init.cpp:257:33: required from here
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:745:3: error: static assertion failed: YOU_MIXED_DIFFERENT_NUMERIC_TYPES__YOU_NEED_TO_USE_THE_CAST_METHOD_OF_MATRIXBASE_TO_CAST_NUMERIC_TYPES_EXPLICITLY
EIGEN_CHECK_BINARY_COMPATIBILIY(Func,typename ActualDstTypeCleaned::Scalar,typename Src::Scalar);
^
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h: In instantiation of ‘void Eigen::internal::call_assignment_no_alias(Dst&, const Src&, const Func&) [with Dst = Eigen::Matrix<std::complex, -1, -1>; Src = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >; Func = Eigen::internal::assign_op<std::complex >]’:
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:712:27: required from ‘void Eigen::internal::call_assignment(Dst&, const Src&, const Func&, typename Eigen::internal::enable_if<(! Eigen::internal::evaluator_assume_aliasing::value), void*>::type) [with Dst = Eigen::Matrix<std::complex, -1, -1>; Src = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >; Func = Eigen::internal::assign_op<std::complex >; typename Eigen::internal::enable_if<(! Eigen::internal::evaluator_assume_aliasing::value), void*>::type = void*]’
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:693:18: required from ‘void Eigen::internal::call_assignment(Dst&, const Src&) [with Dst = Eigen::Matrix<std::complex, -1, -1>; Src = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >]’
/usr/include/eigen3/Eigen/src/Core/Assign.h:75:28: required from ‘Derived& Eigen::MatrixBase::operator=(const Eigen::EigenBase&) [with OtherDerived = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >; Derived = Eigen::Matrix<std::complex, -1, -1>]’
/usr/include/eigen3/Eigen/src/Core/PlainObjectBase.h:543:22: required from ‘Derived& Eigen::PlainObjectBase::operator=(const Eigen::EigenBase&) [with OtherDerived = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >; Derived = Eigen::Matrix<std::complex, -1, -1>]’
/usr/include/eigen3/Eigen/src/Core/PlainObjectBase.h:521:13: required from ‘Eigen::PlainObjectBase::PlainObjectBase(const Eigen::EigenBase&) [with OtherDerived = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >; Derived = Eigen::Matrix<std::complex, -1, -1>]’
/usr/include/eigen3/Eigen/src/Core/Matrix.h:379:29: required from ‘Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Eigen::EigenBase&) [with OtherDerived = Eigen::DiagonalWrapper<const Eigen::Matrix<double, -1, 1> >; _Scalar = std::complex; int _Rows = -1; int _Cols = -1; int _Options = 0; int _MaxRows = -1; int _MaxCols = -1]’
Operation/pose_regression_init.cpp:256:45: required from here
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:745:3: error: static assertion failed: YOU_MIXED_DIFFERENT_NUMERIC_TYPES__YOU_NEED_TO_USE_THE_CAST_METHOD_OF_MATRIXBASE_TO_CAST_NUMERIC_TYPES_EXPLICITLY
cc1plus: fatal error: can’t open ‘/tmp/ccmTFx4G.s’ for writing: No such file or directory
compilation terminated.
cc1plus: fatal error: can’t open ‘/tmp/ccmTFx4G.s’ for writing: No such file or directory
compilation terminated.
cc1plus: fatal error: can’t open ‘/tmp/ccmTFx4G.s’ for writing: No such file or directory
compilation terminated.
Makefile:2: recipe for target 'all' failed
make: *** [all] Error 1

在https://github.com/chensong1995/HybridPose/blob/master/lib/datasets/linemod.py#L187 这样定义t的ground truth。用186行直接读取的不行吗？抱歉我没有看懂这里的算法，是否这样做可以使之更容易收敛，请指教一下

1.How to improve the effect of serious occlusion? What are the suggestions？
2.Is the network effective for non rigid objects?
3.What is the effect of multi-target detection in complex scenes？
Thank you very much.

Hi @chensong1995 ,
I'd like to train from scratch and for testing purposes do something like this.
$ python src/train_core.py --batch_size 1 --n_epochs 2 --object_name cat --load_dir None

But this outputs:
-> print('Could not restore session properly, check the load_dir')
(Pdb)

If I add the parameter --load_dir None it sets load_dir to the string 'None' and is not None in terms of this line:
if args.load_dir is not None:

However, if I modify it to something like: if args.load_dir != 'None':
it seems to work. At least its training the ResNet networks.

After the nets are trained there is another error in the trainer.generate_data().
I think it happens in this line, where the pr_para, pi_para = self.search_para(...) function gets called.

It outputs following in the end:

Epoch: [1][946/949]	Time: 0.063 (0.066)	Sym: 1.4552 (4.4397)	Mask: 0.0115 (0.0254)	Pts: 0.0234 (0.0563)	Graph: 4.4105 (7.8646)	Total: 0.8322 (1.8185)
Epoch: [1][947/949]	Time: 0.062 (0.066)	Sym: 3.8132 (4.4390)	Mask: 0.0290 (0.0254)	Pts: 0.0578 (0.0563)	Graph: 3.8165 (7.8603)	Total: 1.3695 (1.8181)
Epoch: [1][948/949]	Time: 0.063 (0.066)	Sym: 1.5503 (4.4360)	Mask: 0.0092 (0.0253)	Pts: 0.0182 (0.0562)	Graph: 3.4466 (7.8556)	Total: 0.6913 (1.8169)
python: eigen/Eigen/src/Core/DenseCoeffsBase.h:410: Eigen::DenseCoeffsBase<Derived, 1>::Scalar& Eigen::DenseCoeffsBase<Derived, 1>::operator[](Eigen::Index) [with Derived = Eigen::Matrix<double, -1, 1>; Eigen::DenseCoeffsBase<Derived, 1>::Scalar = double; Eigen::Index = long int]: Assertion `index >= 0 && index < size()' failed.
Aborted (core dumped)

If I run it with the --load_dir set to a saved model it runs through the training and the trainer.generate_data().

$ python src/train_core.py --batch_size 1 --n_epochs 501 --object_name cat --load_dir /home/nixi/Projects/HybridPose_custom/data/saved_weights/occlusion_linemod/cat/checkpoints/0.02/499

That will output:
saved
So that means for me that the regressor can access the eigen library and the $LD_LIBRARY path is setup correctly.

Do I miss something?
Any idea whats going on?

Thanks and keep up the good work!

Hi, I have a question about the training strategy of HybridPose.

After reading your codes, I found that the network is also trained on the OCCLUSION LineMOD dataset. However, prior methods such as PVNet and PoseCNN use the LineMOD dataset for training and the OCCLUSION LineMOD dataset for only testing.

I believe that the training on the OCCLUSION LineMOD dataset can cause an unfair comparison. What do you think about this point?

Hello,

I was wondering if you could expand more on the intructions for the Training portion where you said:

"Please set the arguments in src/train_core.py execute the following command (note that we need to set LD_LIBRARY_PATH for the pose regressor):
LD_LIBRARY_PATH=lib/regressor:$LD_LIBRARY_PATH python src/train_core.py"

Could you perhaps give a more step-by-step description on what to do? I am new to all of this and I've been gradually learning.

Thank You

Which OS did you use for this project?

I suspect the problem is caused by inconsistent behaviors of rand_val() in pose_regression_parameter_search.cpp on different platforms.

This means readers may not be able to reproduce our results exactly.

Hi Chen,

Your work is impressive. Can you provide an example that depicts how to load the pre-trained network and execute it in a given RGB image? I tried to discover from train.py and evaluate.py, but I failed.

Hello,

First of all, thank you for your great work. We are writing to you and not the original author of SymSeg since you have recently used the library successfully.

We are trying to generate ground truth labels for synthetic data but have run into some issues when using SymSeg to generate the symmetry correspondences. In particular, objects which are clearly symmetric across some planes do not have any symmetries detected, perhaps due to the downsampling of the cloud?

Did you have to modify or tweak any of the code/config parameters before computing the symmetries to make it work?

Furthermore, we also get segmentation faults (bad alloc) when trying to run any of the actual symmetry segmentation scripts (only symmetry detection works). Do you recall which versions of the dependencies of SymSeg you used?

Thank you

Hello
I was wondering if you could provide a bit more documentation on how to train on own dataset. Say I want to use this project for getting the 6D pose of chairs. Could you clarify:

• what format does my dataset need?
• what about the labeling?
• where to place what in the folder structure?
• finally, how do we run the training and then evaluate results (commands to execute)?

Loved your project work. I'm a huge fan and I am excited to learn from your steps!

Is there a way to visualize the training curve?

When running the code, it was suggested that several representations mentioned in the paper could not be generated.

The predicted symmetry correspondences is only random points not lines (numbers are too small they dont form a line). Meanwhile the graph and keypoints are predicted correctly. Why does that happen?

你好，首先感谢你优秀的工作，使我初步入门6d位姿估计便受到极大的启发。目前，我尝试自己制作一个类似于occlusionlinemod的数据集。想请教一下标签中的pose真值如何获得？同时，关于mask的制作，我是否可以使用类似于lableme的语义分割标注工具制作？其他位姿的中间表示，Keypoints 和Symmetry的推断来源于什么原始信息？（抱歉我虽然看了链接，但怕自己理解有误）。我不知是否还漏掉了其他标签，还请不吝赐教。感谢！

hello~
I notice that in the paper there are three network to predict 3 kinds of intermidiate representations.
but in the code, I only see there is one network which is the same as PVNet's, so I want to know if my understanding is true: use the same network in PVNet to predict two more intermidiate representations: egde vetors and symmetry correspondences? and add two correspoding loss to these two task?
Another question is where can I change the training-test split in the code?
wish your reply~thanks

The aruco markers (QR codes )help provide the Rt matrix. If they are used in the training data, do they need to be also present during inference?

What would be the best way to acquire the Rt (ground truth pose matrix) with/without the qr codes in your personal opinion ?

The following packages will be SUPERSEDED by a higher-priority channel:

ca-certificates pkgs/main --> anaconda
certifi pkgs/main --> anaconda
openssl pkgs/main --> anaconda
setuptools pkgs/main::setuptools-47.3.1-py37_0 --> anaconda::setuptools-39.1.0-py37_0

how to solve this issue , after the command conda install -y -q --name hybridpose -c pytorch -c anaconda -c conda-forge -c pypi --file requirements.txt it gives this statement.

I download eigen from http://eigen.tuxfamily.org/index.php?title=Main_Page
and copy HybridPose/lib/regressor/eigen

when I do make in HybridPose/lib/regressor, it show this:

Operation/pose_regression_parameter_search.cpp:365:31: error: ‘Eigen::VectorXd {aka class Eigen::Matrix<double, -1, 1>}’ has no member named ‘end’
       sort(temp.begin(), temp.end());

Maybe my eigen is error?

Thank you for sharing your research.
I want to use T-less datasets.
http://cmp.felk.cvut.cz/t-less/

I have some difficulties trying to fit your data-format.

Q1.
How do you create a "cor{}.npy" file in "correspondences" folder?
Is it created by running the project? (https://github.com/aecins/symseg)
https://github.com/chensong1995/HybridPose/blob/master/lib/datasets/linemod.py#L211

Q2.
What do the point and normal values of the "symmetries.txt" file in the "symmetries" folder mean and how do you create it?
https://github.com/chensong1995/HybridPose/blob/master/lib/datasets/linemod.py#L212

Q3.
Are pts2d and pts3d the same as FPS_2d and FPS_3d provided by pvnet?
https://github.com/chensong1995/HybridPose/blob/master/lib/datasets/linemod.py#L206

pvnet fps_3d and fps_2d:
https://github.com/zju3dv/clean-pvnet/blob/master/lib/datasets/linemod/linemod_to_coco.py#L270

Q4.
Is the unit of translation in meters?

Thank you in advance for your reply.

If you have a code that makes datasets-format, please share it with us.

Regards

David

你好；
我在运行train_core.py时报错；
/home/jzy/anaconda3/envs/HybridPose-master/bin/python /home/jzy/桌面/jzy/HybridPose-master/src/train_core.py
number of model parameters: 12959563
Traceback (most recent call last):
File "/home/jzy/桌面/jzy/HybridPose-master/src/train_core.py", line 125, in
trainer.train(epoch)
File "/home/jzy/桌面/jzy/HybridPose-master/trainers/coretrainer.py", line 37, in train
for i_batch, batch in enumerate(self.train_loader):
File "/home/jzy/anaconda3/envs/HybridPose-master/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in next
return self._process_data(data)
File "/home/jzy/anaconda3/envs/HybridPose-master/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/jzy/anaconda3/envs/HybridPose-master/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 0.
请问；是需要将parser.add_argument('--load_dir', type=str, default=None)中的‘None’换成自己的路径么，如果是的话，要换成的是什么呢？

你好,请问可以分享一下6D位姿可视化的代码吗?

Thank you for your valuable research and code.

Do you only use the original Linemod image in your paper? Is offline rendering or online data augmentation used? If so, where can I find the code for rendering or data augmentation

Thank you for sharing your excellent research.

I'm going to run train 1 and then test 1.
However, the ADD value cannot be output because there is no prediction_file to proceed with the test.

So I want you to tell me how to save and get prediction_file file.
https://github.com/chensong1995/HybridPose/blob/master/src/evaluate.py#L12

https://github.com/chensong1995/HybridPose/blob/master/src/evaluate.py#L65

Thank you very much.

Regards,
David.

Hello @chensong1995 ,
thanks for your research, making this code available and answering all those questions from the community!

I can't figure out how to run (just) inference on HybridPose.

The prediction part is not the problem.
I've downloaded the pretrained weights, modified the forward() so I can compute
sym_cor_pred, mask_pred, pts2d_map_pred, graph_pred = model(img)

I struggle with the regression module.
R_pred, t_pred, R_init, t_init = self.regress_pose(...)

I think I can:

• pass regressor and predictions
• pass camera_intrinsic to calculate K_inv.
• pass the estimated pred matrixes of the prediction network.

Questions:

1. Is the calculation of pr_para and pi_para just necessary one time? So I could save them and pass a constant when running inference.

2. What is batch['pts3d'][i] and batch['normal'][i] doing in regress_pose()? They are from the dataset and should not matter during inference. But I can't follow your code along and don't know what to pass.

Thanks in advance!

According to Dataset set-up section in README.md, symmetry labels were generated using symseg.
It is not clearly stated in README.md (😭), but specifically what needs to be done is the generation of symmetries.txt.

If you follow the setup procedure in README.md, these symmetries.txt files are downloaded automatically when you execute:

python data/download_linemod.py

and can be found at data/linemod/symmetries/*/symmetries.txt.

Since I'm making my own dataset with my own classes, I need to generate data/linemod/keypoints/*/keypoints_3d.npy and data/linemod/symmetries/*/symmetries.txt for each of my classes.
I was able to generate keypoints_3d.npy using the method shown below:

from argparse import ArgumentParser
from lib.csrc.fps.fps_utils import farthest_point_sampling
from lib.utils.linemod.opengl_renderer import OpenGLRenderer
import numpy as np

def parse_args():
    parser = ArgumentParser()
    # training
    parser.add_argument('--ply_path', type=str, required=True)
    parser.add_argument('--save_path', type=str, default='model_kpts3d.npy')
    parser.add_argument('--num_keypoints', type=int, default=8)
    args = parser.parse_args()
    return args

args = parse_args()
renderer = OpenGLRenderer(args.ply_path)
sampled_kpts3d = farthest_point_sampling(pts=renderer.model['pts']/1000, sn=args.num_keypoints, init_center=True)
np.save(args.save_path, sampled_kpts3d)

Next I need to generate symmetries.txt, so I setup symseg and confirmed that the example provided is working correctly.

However, I have no idea what to do from here. I was expecting to see some sort of API with options for creating a symmetries.txt dump file, but I can't seem to find that anywhere. (Is there even an API for this tool at all?)

I need some clarification on how specifically the symmetries.txt files were created using symseg. Also, if no symmetries are found, does that mean it is impossible to correctly create labels for training the HybridPose model?

Hi Chen,
thanks for sharing the work. I have some questions while reading your code:

1. I download the augmented labels and see the valid poses you provided. The number of the valid poses is a lot less than the number of gt poses (1214) provided by the original Occlusion LineMOD. I wonder how you got these valid poses and why you use these valid poses for evaluation instead of the gt poses provided by the original Occlusion LineMOD?

2. In setup_loaders(), I see you randomly select 80% of data in Occlusion LineMOD as the training set and the other 20% as the test set. In generate_data() you search for parameters with the first 20 samples of the test set and evaluate with the remaining samples in the test set. Am I right? Please correct me if my understanding is wrong.

I will be looking forward to your reply.

Hello

I am curious about the difference between pose_init (R_init & t_init) and pose_pred (R_pred & t_pred).
In case of pose_pred, did you make the prediction using symmetric and graph?
Did I understand correctly?

Please explain in detail.

Thank you.

David.

Hello Chen,
thanks for the inspiring paper.
I wonder if there is any additional material that you can recommend to understand the formula described in the paper. I found it very interesting how you combined all the inputs to contains and optimize the variables etc.. However, there is a lot I do not understand. So my question is where you got inspired to this mathematical approach and how can I learn it?
Thanks a lot,
Matthes

Hi, I wanted to see how the model is working so I thought I would download the pre-trained weights and give some images as input and see the 3D bounding box as output. But I can see that there are different weights for different classes. Why there are different weights? I am new in this field. As far as I know, we require only one pre-trained weight for all the classes? Please help me with this. Sorry if I am missing some obvious things.

Hello,

First of all, thank you for sharing your work.

I am writing because I have managed to deploy and train Hybridpose on my local machine, but when I launch it via remote Pycharm on a shared GPU server, I get the following message during test phase (in learning phase there is no problem):

GPUassert: a PTX JIT compilation failed ./src/ransac_voting_kernel.cu 83

On both machines the code is obviously the same and the anaconda environment is a copy from my local machine (through a .yml file).

The only difference I see is that nvidia drivers are different on my local machine (v440.33.01) and on the remote machine (v418.87.00),but I understand that there should be no problem.

Any idea where the problem might be?

Thank you very much!

Hello,

Thank you for your great work. I have a question regarding the architecture of neural network. You disable the last global average pooling layer that is normally used in ResNet. Why is this? The reasoning behind this does not appear to be mentioned in either PVNet or your work.

Regards,

hello~ 请问测试时parser.add_argument('--prediction_file', type=str, default='output/occlusion_linemod/test_set_ape_40.npy')中的test_set_ape_40.npy是什么作用呢，我输入默认命令行python src/evaluate.py会报以上的错误，但是不知道test_set_ape_40.npy在哪，并且在测试里有什么作用。
期待您的回复~谢谢！

Hello, I wanted to predict the pose of my object(pole) .
And then, many ways are based on depth image or point cloud data.
Unfortunately, I couldn't get those data. I just can get single images captured by normal camera.
I tried to use 3D object detection at first, but failed several times.
I found 6D pose estimation possible, I think.
And then, I want to know whether it is possible to predict the 6D pose of my object(pole) from one single image.
If yes, how can I train them?
I have more than 10000 images.
Thank you in advance.

HI, I am very interested in your work and trying to play with the code. However, I encounter an error when I was trying to run the train_core.py. Would you like to take a look?

Traceback (most recent call last):
  File "src/train_core.py", line 105, in <module>
    trainer.generate_data()
  File "/home/robot/HybridPose/trainers/coretrainer.py", line 558, in generate_data
    regressor = load_wrapper()
  File "/home/robot/HybridPose/lib/regressor/regressor.py", line 13, in load_wrapper
    regressor = CDLL('wrapper.so')
  File "/home/robot/anaconda3/envs/hybridpose/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: wrapper.so: cannot open shared object file: No such file or directory

How long does one epoch takes? it is very slow on my dataset, about 1 hour for every epoch. is that normal?

When I use the pretrain weights you provided,I find that the final rotation error is always around 1.5.I'am refused about it.
Another problem is how to achieve the same result as your paper.