tum-vision / mono_dataset_code

Code for Monocular Visual Odometry Dataset - https://vision.cs.tum.edu/data/datasets/mono-dataset

Install

1. Install Eigen & OpenCV (if you don't have it):

sudo apt-get install libeigen3-dev libopencv-dev

2. install ziplib:

sudo apt-get install zlib1g-dev
cd thirdparty
tar -zxvf libzip-1.1.1.tar.gz
cd libzip-1.1.1/
./configure
make
sudo make install
sudo cp lib/zipconf.h /usr/local/include/zipconf.h   # (no idea why that is needed).

3. install aruco marker detection (optional - only required for vignette calibration):

see eg here:

http://maztories.blogspot.de/2013/07/installing-aruco-augmented-reality.html 

tested with version 1.3.0. which is included in /thirdparty.

4. Build

cmake . && make

Usage: C++ code

playbackDataset: read images, photometric undistortion & rectification.

Shows images of a dataset. Meant as example code regarding how to read the dataset. Run with (and replace X with the location of the dataset. Mind the trailing slash):

./playDataset X/sequence_01/  

optionally, the calibration is used for

• rectification ( r )
• response function inversion ( g )
• vignette removal ( v )
• removal of over-exposed (white) images. ( o ).

Pressing the respective key toggles the option. See code for details.

responseCalib: calibrate response function.

Performs photometric calibration from a set of images, showing the exact same scene at different exposures. Run with (and replace X with the location of the dataset. Mind the trailing slash):

./responseCalib X/CalibrationDatasets/narrow_sweep1/"

outputs some intermediate results, and pcalib.txt containing the calibrated inverse response function to ./photoCalibResult. See code for details.

vignetteCalib: calibrate vignette.

Performs photometric calibration from a set of images, showing a flat surface with an ARMarker. Run with (and replace X with the location of the dataset. Mind the trailing slash):

./vignetteCalib X/CalibrationDatasets/narrow_vignette/"

outputs some intermediate results, and vignette.png (16-bit png) containing the calibrated vignette function. vignetteSmoothed.png is a slightly smoothed version, mainly to remove the black borders (pixels at the border are never observed). See code for details. WARNING: requires a lot of Memory (16GB ram for 1000 input images)! Can easily be changed at the cost of slightly slower runtime... you'll have to do that yourself though.

Usage: Matlab evaluation code

Implements Sim(3) alignment of a tracked trajectory to the ground-truth segments, and subsequent computation of the different error values. See MatlabEvaluationCode/Example.m for an example, and some documentation regarding the computed values. Further, we include example results computed with DSO for all 50 sequences.

License

The code provided in this repository is licensed under the BSD license.

mono_dataset_code's Issues

playDataset can't read calibration file

There seems to be a bug with the camera calibration reader in which no float is correctly read. Here's my output, using the sequence 45

Playback dataset /home/alejandro/Downloads/sequence_45/!
Load Dataset /home/alejandro/Downloads/sequence_45/: found 3000 files in folder /images; assuming that all images are there.
0.535719308086809       0.669566858850269       0.493248545285398       0.500408664348414       0.897966326944875
inputCalibration before
0
0
0
0
0
inputCalibration after
0
0
0
0
0
Failed to read camera calibration (invalid format?)
Calibration file: /home/alejandro/Downloads/sequence_45/camera.txt
Reading Photometric Calibration from file /home/alejandro/Downloads/sequence_45/pcalib.txt
Reading Vignette Image from /home/alejandro/Downloads/sequence_45/vignette.png
PhotometricUndistorter: Invalid vignette image size! got 1280 x 1024, expected 0 x 0. Set vignette to 1.
Dataset /home/alejandro/Downloads/sequence_45/: Got 3000 files!
Rectified Images: -1 x 0. K:
2.10578e-08 4.56277e-41 1.41933e-37
4.56277e-41 1.41933e-37           0
2.10578e-08           0           0

Original Images: 0 x 0. omega=0,000000 K:
0 0 0
0 0 0
0 0 0

ERROR: expected cv-mat to have dimensions 0 x 0; found 1280 x 1024 (image /home/alejandro/Downloads/sequence_45/images/00000.jpg)!
Segmentation fault (core dumped)

I added those lines for debugging purposes. Seems like
std::sscanf(l1.c_str(),"%f %f %f %f %f", &inputCalibration[0], &inputCalibration[1], &inputCalibration[2], &inputCalibration[3], &inputCalibration[4]);
at FOVundistorter.cpp isn't parsing the calibration file correctly. I've tried with other sequences to no avail.

Vignette calibration problem

I am running vignetteCalib on my own dataset - just 30 images to test the approach.
The program produces only black vignette and following output:

Load Dataset samples_vignette3/: found 30 files in folder /images; assuming that all images are there.
Input resolution: 856 480
Input Calibration (fx fy cx cy): 338.125397 194.108505 269.381378 83.858086 1.000000
NO RECTIFICATION
Reading Photometric Calibration from file samples_vignette3/pcalib.txt
Reading Vignette Image from samples_vignette3/vignette.png
PhotometricUndistorter: Invalid vignette image size! got 0 x 0, expected 856 x 480. Set vignette to 1.
Dataset samples_vignette3/: Got 30 files!
SEQUENCE NAME: samples_vignette3/!
plane image values 10000000000.000000 - -10000000000.000000!
0.000000 residual terms => -nan
0.000000 residual terms => -nan
plane image values 10000000000.000000 - -10000000000.000000!
0.000000 residual terms => -nan
0.000000 residual terms => -nan
...

the output is in the same fashion until the end.

Any idea what might be wrong with my dataset?
Is it possible that it does not work because it is not composed from video capture? (I can only get exposure times for the third column of times.txt when shooting images one by one with the camera)

Code Licensing?

What license is the code and data available under?

About times.txt format

Hi,

What is the time unit in times.txt (third column)?

How to calibrate camera?

I use example from OpenCV for camera calibration. After calibrate matrix check it on learned images. All looks fine. Also get 5 values for distortion, but not use anywhere.

After this try to adapt to yours format and check on playDataset example.

double fx = cameraMatrix.at<double>(0, 0) / (double)imageSize.width;
double fy = cameraMatrix.at<double>(1, 1) / (double)imageSize.height;
double cx = (cameraMatrix.at<double>(0, 2) + 0.5) / (double)imageSize.width;
double cy = (cameraMatrix.at<double>(1, 2) + 0.5) / (double)imageSize.height;

Camera matrix looks like:

0.820361393435219	0.461474391295798	0.495088499159556	0.529959721497076	0
1280 720
crop
1280 720

After undistortion nothing changed. If I set omega to some value, undistortion apply but with strong skew of image. So how you calculate values for camera.txt file?

create tum dataset for my video

I want to convert my own video data into a folder structure like TUM benchmarks. So you want to create times.txt, camera.txt, statistic.txt and times.txt. how do I do this automatically? or what do the columns in statistic.txt and times.txt mean?

fx fy cx cy k1 k2 p1 p2 inputWidth inputHeight "crop" / "full" / "none" / "e1 e2 e3 e4 0" outputWidth outputHeight
0.349153000000000 0.436593000000000 0.493140000000000 0.499021000000000 0.933271000000000
???

What is the unit of fx an fy?

In my case:
Camera.fx: 1138.042594
Camera.fy: 1152.150891
Camera.cx: 314.573090
Camera.cy: 273.959896
unit is pixel.

What about the TUM mono dataset?

Good RMSE values

I was wondering what are typical RMSE values on a good calibration.

Calibrate response receives infinite results

Hi

I'm trying to calibrate the response of my camera. When I run the program, I found the results as follow,

It shows the irradiance and inverse response goes to infinite.
My guess is it related to the mean irradiance of all images, which range between [2.5, 182.6]. However, the exposure I set is between [0.06ms 200ms], with 167 images and exposure in multiplicative increments of 1.05. It is definitely overexposed at large expo time, and visually black at low expo time. I'm not sure what's wrong with this image data.

Meanwhile, the following points are especially noticed and ensured,

• a constant scene with not moving objects and change light.
• the compiled program worked correctly on the "narrow_sweep3" dataset.

Please let me if you have any advice.

Thanks

camera ISP affected the manual calibration

We are doing the manual photometric calibration on our embedded system, so far we still have two problems:

1. Change camera exposure time costs 0.5 hour to execute
2. The ISP for camera will automatically adjust the exposure time if it is over exposure, and also ISP will adjust other parameters to affect Vignette calibration.

How shall we handle these problems? Please help to advise, thanks

Do not know how undistort an image

I am sorry I did not get it how to undistort the images using this code. I did not find the distortion coefficients either. So, where can I find it? Like k1, k2, k3, p1 and p2?

question about the evaluation result of dso and orbslam

hello
I've read your paper "A Photometrically Calibrated Benchmark For Monocular Visual Odometry".
In section 3.2, figure 8 you compared the result of dso and orbslam by 500 runs.
The figure is upload below:

However i have a question here: from the figure we can see that as the number of runs increases, the error increases correspondingly, but when the number of run increases from approximately 480 to 500, the error increases so fast. I think it's not so feasible or may i just can't explain this phenomenon.
In the paper there are a lot of figures like this having the same result. Could you please explain this phenomenon? Thank you so much!
best
chenchr

pcalib.txt and vignette.png generation on TUM VO dataset

Hello,

I am trying to build pcalib.txt and vignette.png on one of the sequences of TUM dataset. Using ./responseCalib I am able to generate pcalib.txt on one of the sequence but it isn't the same as the original pcalib.txt. I am not able to understand why is it different. Not only I get non-increasing values also all the values lie in 100...255 range.

Also I am not able to generate vignette.png using ./vignetteCalib on the same sequence using the original pcalib.txt since the pcalib generated using ./responseCalib is not correct.
(PS: I have aruco already installed and tried creating marker board example after installation. That works fine.)

I also encounter an error when I run ./playDataset on TUM dataset sequence. The error looks like

Has anyone encountered some similar problem or have some solution to this?

Thank you!

record different dataset and get different calibration result

Hello，
I recorded several datasets, but the calibration results of these datasets are different. I don't know which result is right. How to solve this?

response function calibration with own dataset

Hello, @NikolausDemmel i want to do response function calibration with my webcam which is auto exposure and a Mono industrial camera with manual exposure setting. How can i generate dataset with different exposure time as suggested in algorithm.
Thanks in advance.

Can I save calibrated image?

I want to save calibrated images, but it seems the playDataset is just an image reader?

Camera response calibration

Hi,
I've a problem with calculating inverse camera response function on my own data set.
the ICRF doesn't appear to be monotonic (or normal) at all, knowing that I've tested the code on different data sets that I've created on my own.
I've attached a sample of (G) that appears to me in the following link and times.txt file.
range of exposure time in my camera is 0.01 ms up to 100 ms.
CRF : https://ibb.co/eSj2A6
times.txt: https://file.io/XZMlAm

Checkerboard size

Where is it possible to get the checkerboard sizes for sequences calib_narrow_checkerboard1/2 and wide_checkerboard1/2 ?
Nothing on the webpage and the paper doesn't seem to contain any information.

Unable to run playDataset with mono_dataset sequences

Hello
I successfully compiled " mono dataset " but when i am trying to run it with the command
./playDataset X/sequence_01 but getting the error that :-

bash: ./playDataset: No such file or directory

Can anyone tell me where this file is ,i am unable to find it.
Thanks .

Inv. response function U is not monotonic

Hello there, thanks for open-source.
I have a question that the inverse response function U is not monotonic based on my dataset (more than 1000 images, static scene and different exposure time from 0.05ms to 20ms). You said in this case, "it" should be smoothed or perturbed on your paper, and what is "it"?
I have modified some code like "G[255]" to "Gmax"(which is the max value in the G[]), is there some other codes I need to modified?
I'm sure that there are many different objects in the static scene, and the exposure time are incremental.

Best wishes.

Camera response calibration

Hi, we have some problems with reproducing the experiment. For the inverse response calibration, it doesn't appear to be monotonic, so we think that some errors exist in our data. Therefore, we want to ask some help for capturing test data:
1)There are some image algorithms for improving the image quality in our camera,like AWB, Positional Gain Adjustment, Adaptive Local Tone Mapping ,Color Correcton and Aperture Correction. Except the gamma correction , whether the above mentioned algorithoms should be disabled?Or have you some suggestions about the camera parameter set-up, which one should be enable or which one should be disable?
2) For recording the images, range of exposure in our camera is 0.19ms up to 31ms, but we can't get the overexposed images, if we haven't the overexposed images, whether it's a factor influencing the response funtion generation?
3)Our camera is rolling shutter, does it matter?
Please help.Thanks

Remove unnecessary null pointer checks

Extra null pointer checks are not needed in functions like the following.

• PhotometricUndistorter
• UndistorterFOV

Unable to run playDataset with mono_dataset sequences

Hello.
I compiled the mono_dataset_code successfully. Now I want to run ./playDataset for the sequences available in mono_dataset. ( https://vision.cs.tum.edu/data/datasets/mono-dataset )
I get some different errors while trying to run it on some of the sequences.
For the first sequence, I run:
./playDataset /home/dso/sequence_01/

It shows this message:

Playback dataset /home/dso/sequence_01/!
Load Dataset /home/dso/sequence_01/: found 4757 files in folder /images; assuming that all images are there.
Failed to read camera calibration (invalid format?)
Calibration file: /home/dso/sequence_01/camera.txt
Reading Photometric Calibration from file /home/dso/sequence_01/pcalib.txt
Reading Vignette Image from /home/dso/sequence_01/vignette.png
PhotometricUndistorter: Invalid vignette image size! got 1280 x 1024, expected 9 x 0. Set vignette to 1.
Dataset /home/dso/sequence_01/: Got 4757 files!
Rectified Images: 21199456 x 0. K:
-8.12748e-30 0 3.53393e-38
4.58953e-41 3.57133e-38 0
3.59111e-38 0 -nan
_
Original Images: 9 x 0. omega=0,000000 K:
0 8.96831e-44 0
1.06809e-05 0 3.5766e-38
2.89901e+29 6.72623e-44 0

ERROR: expected cv-mat to have dimensions 9 x 0; found 1280 x 1024 (image /home/dso/sequence_01/images/00000.jpg)!
Segmentation fault (core dumped)

Does anyone know what the problem is?
Thanks a lot.

responseCalib error : buffer still too small. abort.

I tried to make pcalib.txt for my custom data. I prepared images.zip, camera.txt, times.txt.

Data:
images.zip

camera.txt
RadTan 0.39296 0.70102 0.49574 0.51231 0.00018 -0.01646 0.00351 0.000281
1280 720
crop
1280 720

times.txt
00001 1659702911.5288742000 16.7
00002 1659702911.5455742000 16.7
00003 1659702911.5622742000 16.7
00004 1659702911.5789742000 16.7
00005 1659702911.5956743000 16.7
00006 1659702911.6123743000 16.7
00007 1659702911.6290743000 16.7
00008 1659702911.6457744000 16.7
00009 1659702911.6624744000 16.7
00010 1659702911.6791744000 16.7
00011 1659702911.6958745000 16.7
00012 1659702911.7125745000 16.7
00013 1659702911.7292745000 16.7
00014 1659702911.7459745000 16.7
....

input:
~/mono_dataset_code/bin$ ./responseCalib ../dataset_0805/

output:
Load Dataset ../dataset_0805/: found no in folder /images; assuming that images are zipped.
got 587 entries and 587 files from zipfile!
Failed to read camera calibration (invalid format?)
Calibration file: ../dataset_0805/camera.txt
Reading Photometric Calibration from file ../dataset_0805/pcalib.txt
PhotometricUndistorter: Could not open file!
Dataset ../dataset_0805/: Got 587 files!
read 10000/10000 bytes for file 00001.png. increase buffer!!
buffer still to small (read 100000/100000). abort.

I don't know how to solve this problem.

Uninitialized values in undistorter

Vignette calibration crashed with this error:

OpenCV Error: Assertion failed (s >= 0) in setSize, file /tmp/binarydeb/ros-kinetic-opencv3-3.2.0/modules/core/src/matrix.cpp, line 307 terminate called after throwing an instance of 'cv::Exception' what(): /tmp/binarydeb/ros-kinetic-opencv3-3.2.0/modules/core/src/matrix.cpp:307: error: (-215) s >= 0 in function setSize

I was using following camera calibration:

395.006308 404.392711 314.697857 174.704342 0.0
856 480
none
856 480

The problem is in FOVUndistorter.cpp on the line 99. Undistorter returns and never reads output width and height which later results in error.

I suggest simply initializing out_width and out_height with values of in_width and in_height respectively.
Patch is attached.
patch.txt

Improve exception safety with smart pointers

Would you like to wrap any pointer data members with the class template “std::unique_ptr”?

broken links for "Calibration sequences"

The download links to the calibration dataset under the section "Calibration sequences" of the mono-dataset homepage (http://vision.in.tum.de/data/datasets/mono-dataset?redirect=1) are all broken, except for the 'All calibration sequences.' Could it be fix?

false result of responseCalib

pcalib.txt

Hi!
I used your program “responseCalib” to test my data and get the response function as shown in the figure:
I have a question, why is the beginning of the first few data is -nan?
and the picture is missing a part of the left.
my data is:
Exposure Time increased from 0.1ms to 20ms. And each collection of 10 pictures, exposure time increased by 0.1ms. so I save 2000 pictures with Exposure Time.
( My camera can only get 0.1ms accuracy )
Thank you very much!