hku-mars / livox_camera_calib Goto Github PK
View Code? Open in Web Editor NEWThis repository is used for automatic calibration between high resolution LiDAR and camera in targetless scenes.
License: GNU General Public License v2.0
This repository is used for automatic calibration between high resolution LiDAR and camera in targetless scenes.
License: GNU General Public License v2.0
Hello,
I tried with non rectified images and rectified images. Calibration output is much better with non rectified images. Is the code is expecting distorted images? Are you running rectification in the pipeline anywhere?
Regards
Thanks for your impressive work. It works on my setup Livox mid70 + Realsense L515 when use_rough_calib is True with the default initial extrinsic matrix. However, I still have some questions.
Most edges match well. But the edges have some offset at the boundary of the field of view. Is it due to the distortion of the camera? I have used the dist_coeffs from the Realsense API. How do you retrieve the dist_coeffs? Maybe the given coeffs from Realsense are not accurate.
Is there any benefit to setting use_rough_calib as False with a better initial extrinsic matrix? I have done several trials. When I use the default initial extrinsic matrix, the final calibration just has obvious offsets. when I use a better initial extrinsic matrix, the calibration drifts out of the image in the optimization stage, i.e no points in the image anymore.
Thanks for your help in advance.
您好,我正在进行一项研究
请问是否可以分享下 livox+D435setting下点云 对齐到D435双目图像上的一些数据?
万分感谢!
[email protected]
wechat 16600286930
Best!
Xuanyi
在void Calibration::colorCloud函数里有以下代码:
for (size_t i = 0; i < pts_2d.size(); i++) {
if (pts_2d[i].x >= 0 && pts_2d[i].x <= image_cols && pts_2d[i].y >= 0 &&
pts_2d[i].y <= image_rows) {
cv::Scalar color = rgb_img.atcv::Vec3b(pts_2d[i]);
if (color[0] == 0 && color[1] == 0 && color[2] == 0) {
continue;
}
其中cv::Scalar color = rgb_img.atcv::Vec3b(pts_2d[i]);是否应改为cv::Scalar color = rgb_img.atcv::Vec3b(pts_2d[i].y,pts_2d[i].x),
opencv 访问数据看起来应该是这种方式
Hello,
Thanks for sharing such amazing work. I am able to calibrate livox avia and rgb camera really well. I am trying modify fast lio 2 (https://github.com/hku-mars/FAST_LIO) for rgb point cloud. But the getting good results only in first 2 3 frames. After that mapping is not good. I saw your example video even you are using fast lio for SLAM and then colouring it. Can you please throw some light for the same?
Thanks a lot
Hi
I have checked your git repository regarding lidar camera calibration. I am a research intern at my Home university NUST Pakistan. I have an os1 64 spinning lidar available at my lab. I need to calibrate the camera and lidar. Now kindly tell me the precise step I need to follow to get that calibration parameter. I don't know how to use Fast LIO for spinning lidar calibration. kindly outline the steps to follow to get extrinsic parameters with your package using os1 64 lidar. I would very thankful to you for this favor.I have bag file recorded with topic pointcloud2 and hikvision camera image. kindly help me in this regard
We were very impressed with the results seen running your sample datasets, but once we tried using our own data, the extrinsics calibration failed in 99% of tested scenarios. In every case, your tool miscalculates scene edges and shifts the image projection, in some cases, by as much as 50-60% off original estimates.
For example, here is the projection of an ultra-HiRes image onto a cloud generated with the Livox Avia:
And here is the same cloud / image pair filtered through your calibration tool:
We provided the same exact intrinsics and initial extrinsics that were used to create the first projection, hoping that your tool would properly recognize scene edges and perfect / optimize the calibration. Instead, it identifies edges far away from their origin and attempts to shift the projection to match them. This results in a completely inaccurate calibration.
Here is the planner cloud:
And the residuals seem to show that the lines are properly detected:
Nonetheless, the optimization shifts everything, producing the result I have posted above. If there are any settings you could recommend that might improve our result, please let me know. I presume that the reason your sample datasets work better is that they are based on scenes with very clean, broad lines, such as those found in artificial structures. We even tried indoor scenes with Aruco / checkerboard plates placed in various locations, and your tool does not recognize any of the lines in these plates. It completely ignores the plates and, as with the example above, "hunts" for edges far from the original extrinsic origin.
Perhaps this tool is only applicable in industrial / city-centric settings where large buildings or artificial structures are present. We will attempt to re-test in a "cleaner" environment, but the fact that this tool cannot work with even standard Aruco / checkerboard patterns to properly orient itself makes it mostly unusable for the majority of contexts we would require.
感谢您的工作。
我准备录取horizon + realsense的数据,我想把他们都录取成一段简短的bag包,然后转换成相应的pcd和图片,因为帧率不同,如何确认他们是同一帧呢。
I have been searching for a targetless automatic lidar-camera calibration package and I came across your repository, however the info section on README says that Velodyne support is to be added later. I was wondering why is it not possible to calibrate Velodyne and RealSense using this package. Theoretically, I can just modify the corresponding topic names before calibration and it should be able to treat the pointcloud as a sensor_msgs/PointCloud2
object independently (from the hardware). Why is that not possible? I am willing to try it myself in a few days but wanted to consult you before that.
Thank you for your work!
include/lidar_camera_calib.hpp line 514
fill_img.at(y, x) = (up_depth + right_depth) / 2;
should be
fill_img.at(y, x) = (up_depth + down_depth) / 2;
您好,请问这个如果要用自己的相机和雷达的数据如何替换呢,录制图片和点云的那个bag包,没有看懂,我刚开始学这个,所以可能问的您觉得有些简单
#19
您好,我遇到了同样的问题,已经把Data.custom_msg: 1改好了,并确认了路径
感觉点云已经加载进去了但还是会报错
您好,感谢你的出色工作,有一个问题我想请教一下:
从论文中看到你是用点云中提取的直线投影到图像中,然后和图像中提取的直线进行拟合。在代码中看到你是用canny来提取直线的,但是canny计算的是边缘而不是直线,那么为什么不用类似houghline这种方法来提取图像的直线呢?
非常感谢您的分享,有几个问题想请教下您:
再次感谢,期待您的回复
Can mid 100 or 40 use this algorithm for automatic calibration?
I have setup an environment with Ubuntu 18.04 and ROS Melodic. All dependencies have been installed, however, using your sample ROS bag files, I am not able to get past the Lidar edge extraction phase. Here is the terminal output:
Camera Matrix:
[1364.45, 0, 958.327;
0, 1366.46, 535.074;
0, 0, 1]
Distortion Coeffs:
[0.0958277;
-0.198233;
-0.000147133;
-0.000430056;
0]
Init extrinsic:
[0, -1, 0, 0;
0, 0, -1, 0;
1, 0, 0, 0;
0, 0, 0, 1]
[ INFO] [1617231323.709287053]: Loading the rosbag /home/visionarymind/Documents/scans/livoxcalib/outdoor1.bag
[ INFO] [1617231328.493049027]: Sucessfully load Point Cloud and Image
[ INFO] [1617231328.493564736]: Building Voxel
[ INFO] [1617231357.841487217]: Extracting Lidar Edge
[lidar_camera_calib-2] process has died [pid 2586, exit code -11, cmd /home/visionarymind/vision_ws/devel/lib/livox_camera_calib/lidar_camera_calib /home/visionarymind/vision_ws/src/livox_camera_calib/config/camera.yaml /home/visionarymind/vision_ws/src/livox_camera_calib/config/config_outdoor.yaml /home/visionarymind/Documents/scans/livoxcalib/outdoor1.bag /home/visionarymind/vision_ws/src/livox_camera_calib/result __name:=lidar_camera_calib __log:=/home/visionarymind/.ros/log/2c3b8fe2-9274-11eb-9545-000c29df74c2/lidar_camera_calib-2.log].
log file: /home/visionarymind/.ros/log/2c3b8fe2-9274-11eb-9545-000c29df74c2/lidar_camera_calib-2*.log
Would you have any advice on how to correct this problem?
demo数据的链接打不开,能否给个百度网盘的链接
T xd = xo * distortion + (distorT[2] * xo * yo + distorT[2] * xo * yo) + distorT[3] * (r2 + xo * xo + xo * xo);
T yd = yo * distortion + distorT[2] * xo * yo + distorT[2] * xo * yo + distorT[2] * (r2 + yo * yo + yo * yo);
should be ?
T xd = xo * distortion + (distorT[2] * xo * yo + distorT[2] * xo * yo) + distorT[3] * (r2 + xo * xo + xo * xo);
T yd = yo * distortion + distorT[3] * xo * yo + distorT[3] * xo * yo + distorT[2] * (r2 + yo * yo + yo * yo);
Hi there,
Cool package, hoping to get this to work! I'm using an os1-64. When I launch the calib the Rough Optimization starts and is pretty good. However, when it moves onto the Optimization step, the point cloud is aligned very poorly. It seems when Optimization starts the cloud is rotated by some amount, maybe 90 degrees.
Here are a couple photos. Please note that my camera images do not fit on my monitor so I can only show what I can see:
Any thoughts or ideas would be appreciated. Thank you
What are the run commands etc..
Thank you for continuing to develop this project. It is extremely useful. Currently, this project does not support fisheye calibrations with OpenCV, so I rewrote the code to accommodate such a calibration using this camera model. Rough optimization works almost perfectly, generating a very close calibration:
Unfortunately, when the algorithm enters the optimization stage, it gradually shrinks the point cloud before finalizing the calibration. Here is a snapshot:
I wished to share a larger area of the screen, but OpenCV resizing is not working (nor is imwrite functional). If you would like for me to share our dataset, please let me know where to upload it. In the meantime, if you have any advice on how to prevent this "shrinking" from happening, please let me know.
I have seen this behavior before when using distortion coefficients and was able to clear it up by setting them to [0,0,0,0]. In this case, they are required to properly conform the points to the fisheye image. I have also tried undistorting the images, and the same shrinking behavior is seen.
当我执行roslaunch livox_camera_calib calib_indoor.launch后,终端提示[ INFO] [1625557327.839229229]: Sucessfully load Point Cloud and Image
[ INFO] [1625557327.840642338]: Building Voxel
[ INFO] [1625557347.866017169]: Extracting Lidar Edge
[ INFO] [1625557352.606176671]: Load all data!
[ INFO] [1625557352.678317787]: Finish prepare!
但是等了好久还是没有变化,还是显示initial extrinsic的图片,不知道这是为什么
livox好像不可以直接录制pcd,如果从rosbag里读取每一帧的pcd则点云密度很低,不知道您是怎么处理的,是使用CloudCompare类的软件merge之后保存的吗?
How to record the image msg to rosbag
Hi,
Because of lower version of PCL(1.17), i prepared a docker image(ros:kinetic-ros-core-xenial), with all dependencies. Also placed the respective files on path.
While running i am getting the following error-
[ INFO] [1629189514.066569998]: Loading the rosbag /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag
process[rviz-3]: started with pid [381]
process 381: D-Bus library appears to be incorrectly set up; failed to read machine uuid: UUID file '/etc/machine-id' should contain a hex string of length 32, not length 0, with no other text
See the manual page for dbus-uuidgen to correct this issue.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
[rviz-3] process has died [pid 381, exit code -11, cmd /opt/ros/kinetic/lib/rviz/rviz -d /opt/catkin_ws/src/livox_camera_calib/rviz_cfg/calib.rviz __name:=rviz __log:=/root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/rviz-3.log].
log file: /root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/rviz-3*.log*
[ INFO] [1629189515.516145283]: Sucessfully load Point Cloud and Image
[ INFO] [1629189515.516546482]: Building Voxel
[ INFO] [1629189539.422697927]: Extracting Lidar Edge
[ERROR] [1629189544.520860218]: Can not load image from /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag
[lidar_camera_calib-2] process has died [pid 374, exit code 255, cmd /opt/catkin_ws/devel/lib/livox_camera_calib/lidar_camera_calib /opt/catkin_ws/src/livox_camera_calib/config/camera.yaml /opt/catkin_ws/src/livox_camera_calib/config/config_outdoor.yaml /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag /opt/catkin_ws/src/livox_camera_calib/result __name:=lidar_camera_calib __log:=/root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/lidar_camera_calib-2.log].
log file: /root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/lidar_camera_calib-2*.log*
I am not sure, why process has been died and telling indoor1.bag is not found but it has already started with the same.
Any suggestions ?
Thanks.
English version
Thank you for the open source. This work is very perfect.
However, I have the some question about the code.
First, what's the vpnp. I don't know the Physical mean. Could you provide me reference?
Second, why's the camera frame map into the pixel frame(image frame), you use the (u0-cx)/ fx. According to my knowledge, I would use (u0)/fx - cx.
Thank you very much.
中文版
感謝您提供的開源程式碼,這個程式碼非常的棒
但是我有一些疑問是有關於code的部份
你好,我用的是速腾128线机械LiDAR,想采用此方案进行和相机的联合标定,这样需要的rosbag大概录制多久的?我理解livox是非重复扫描的可能需要录制的rosbag长一点。而机械式LiDAR是重复扫描的,那我这机械LiDAR 录制10帧是不是就可以了?
Lidar: Livox Mid-100
Camera: Flir BlackFly
Rosbag: https://cloud.tsinghua.edu.cn/f/09ceccd8e1c647f59650/?dl=1
Camera_config: https://cloud.tsinghua.edu.cn/f/2687d771b0ea421ea30d/?dl=1
The calibration result seems promising after rough calibration, by collapsed after fine-tune optimization.
你好,请教一个问题,标定的参数中,旋转部分比较准确,但平移部分相差了大概10几厘米,请问是场景选的太远了吗?由于平移部分不准,远处的点对齐的比较好,距离5米以内的点效果比较差
Hi, thank you for your contribution to the society.
When I run your code, I met a problem:
[ INFO] [1626940489.756662588]: Loading the rosbag /home/catkin_ws/src/livox_camera_calib/data/merge.bag
[ INFO] [1626940490.902931052]: Sucessfully load Point Cloud and Image
[ INFO] [1626940490.903897494]: Building Voxel
[ INFO] [1626940508.755300261]: Extracting Lidar Edge
[ INFO] [1626940526.485055384]: Load all data!
[ INFO] [1626940526.904265229]: Finish prepare!
[pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!
Do you know in which function this error happens? So I can check my own data.
Thank you for your time.
English Version
Hello
I have some question about the calibration problem.
I set my vlp-16 and D345i camera as your configuration and I calibrate the extrinsic parameter. However, the value of extrinsic parameter is much different. I think that the problem is the camera intrinsic parameter. I want to ask you how to calibrate your camera intrinsic parameter (I use Matlab)? Or, maybe I deploy myself lab environment is not good?
PS: The attached image which show the color between the pointcloud and image are not matched.
Thank you, bless you have the good day.
中文版本
你好
我想請教一下問題
就是我將光達(vlp-16)與相機(realsense)像你那樣擺設
然後開始進行校正,但我校正出來的值差很多
我推測有可能是相機內參問題
因此想請問你相機內參是如何校正的呢?
目前我是使用matlab
亦或是我的實驗環境擺設有問題呢?
感謝您, 祝您有個美好的一天
您好, 请问你们使用这个方法最后标定出来的外参误差是多少?我在论文里没找到写这部分的内容
Your work is great and inspiring.
I want to put this code together with my SLAM program. Do you think it is feasible?
If possible, I would like to know the specific interface
thank you
What camera is used in this code?
Is there a ros driver for hikrobot MV-CE060-10UC?
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.