Failed to open blickfeld_preprocessed_latest/xxx.bag_matches.json about direct_visual_lidar_calibration

[koide3]

@jeychandar
As @Seekerzero said, for SuperGlue-based alignment, the point cloud needs enough density to generate a LiDAR image with good similarity with the camera image. Try the dynamic points accumulator to densify the point cloud.

@Seekerzero
Thanks always for your help!

[jeychandar]

@Seekerzero @koide3 As you were mentioned earlier I referred the documentation and collected the sample data Up and Down for longer rosbag record since blickfeld LiDAR is repetitive. I tried the dynamic points accumulator to densify the point cloud by using this command rosrun direct_visual_lidar_calibration preprocess blickfield blickfield_preprocessed_check -adv . It really helped to extract the intensity Image and extrinsic matrix. Finally I able to test Lidar_camera_fusion with resulted extrinsic matrix and it is accurate. Once again Thank you for your support! @koide3 and @Seekerzero

[jeychandar]

@Seekerzero @koide3 Hi Sir, I looked through your code and unable to resolve this issue yet. Could you help me out where to check regarding this issue.

[Seekerzero]

Hi jey, could you send your commandline for running the preprocess step to here. Also, an example of your lidar PointCloud2 fileds format would be helpful.

Besides, from the picture you uploaded here, your point cloud is shown with different colors, which is coming from the intensity values, meaning that the intensities values is correctly saved into the ply file.

Thanks.

[jeychandar]

@Seekerzero Thank you for your reply Sir. I ran this command rosrun direct_visual_lidar_calibration preprocess blickfield blickfield_preprocessed_check -av and also tried -i intensity additionally with this command. I attached pointcloud2 format and publishing option below.

[Seekerzero]

Did you get any error like this: "error: failed to determine point intensity channel automatically". If not, the intensity value for each points should be correctly saved into the ply file.

[jeychandar]

I did not get any errors like what you have mentioned above Sir.

[Seekerzero]

So everything should be correct. Like I said in the previous post, if you see the Turbo color from the viewer, the intensity value for each point is correctly loaded and saved. Please also check this .

[jeychandar]

what could be the reason that these images are empty? The preprocess step is failing to return .bag_matches.json. Does it have anything to do with lidar point cloud frame convention difference?

image

[Seekerzero]

The preprocess step should only return one json file, which is calib.json.

Regarding to the intensity image, I would recommend first to check if the values are returned correctly here. You should see something like [0, ... , 0 , index] since you have a empty image. If there are all zeros for the intensities like I guess, you might look into the lidar_proj, otherwise, it could be something related to opencv.

[jeychandar]

@Seekerzero As you mentioned earlier. I cross verified with some other tool called opencalib/ calib_anything to check if the rosbag has intensity recorded. I am able to extract lidar intensity. since this repo direct_visual_calibration tool creates .ply file with intensity, I am able to analyse that lidar_proj have no issues. As you mentioned it could related to opencv I am not sure where could I able to verify that. Could you help me out to resolve this issue?. For your reference I am using blickfeld driver https://git.tu-berlin.de/ecschuetz/ros_assembly/-/blob/main/src/ros_blickfeld_driver_src-v1.4.3/modules/ros_blickfeld_driver_core/src/blickfeld_driver_point_cloud_parser.cpp?ref_type=heads

[Seekerzero]

You should set a break point after the line of code that I linked in the previous post (or you basically just save the intensity values to a files) to see if the values are all zeros here. This is the first thing you should check.
I believe it is not related to the lidar driver since the ply file are successfully generated and verified.

[jeychandar]

@Seekerzero I have saved the intensity values into text files and it returned zeros's.

[Seekerzero]

Thanks, then you shall look into the here to see if the values were properly assigned.

[jeychandar]

@Seekerzero I have attached two images below one for livox(From the rosbag given in this repo) and another for blickfeld sensor. You can notice the frame convention difference. Do you think it impacts the LiDAR projection?.

Livox Sensor:

Blickfeld sensor:

[Seekerzero]

yup, I believe so. Since the projection range is calculated based on here, and given the fov that calculated from your sensor 72 degree, these points might get all filtered.

[jeychandar]

@Seekerzero Yes true it is because of the frame convention. I am able to get lidar intensity image and also intensity values in somewhere in the middle and other values are zero. I tried with this command rosrun direct_visual_lidar_calibration initial_guess_auto blickfield_preprocessed_check

It throws error like this still
loading blickfield_preprocessed_check/2024-06-08-12-48-43.bag.(png|ply)
error: failed to open blickfield_preprocessed_check/2024-06-08-12-48-43.bag_matches.json
Aborted (core dumped)

[Seekerzero]

after run preprocess, you should run
rosrun direct_visual_lidar_calibration find_matches_superglue.py <your preprocessed folder> before you run the initial_guess_auto.

[jeychandar]

@Seekerzero Yes correct working able to get extrinsic matrix thank you for your guidance. I have another question don't you feel there is some offset in superglue generated png? Let me try this extrinsic value first.

[Seekerzero]

superglue might not work if the lidar image does not get enough similarities to the image capture (not enough density). You might want to look into the slam approach for collect and preprocess the data to accumulate the lidar image. Please look into the document example and check that.