i3drobotics / stereo-vision-toolkit

Stereo image processing suite

Stereo Vision Toolkit

Software Website

Latest Release

Download the latest release here

For detailed instructions see the User Guide

Status

About

I3DR's stereo vision toolkit is an application provided for testing of stereo cameras and gathering of 3D data.

You can calibrate stereo cameras, acquire images and perform matching and 3D reconstruction. You can save raw stereo video to capture a scene and then replay it in the software to fine tune matching parameters.

SVTK is under active development. At the moment the software only officially supports the i3DR Deimos and Phobos cameras, but in principle any usb camera pair or pre-rectified stereo video of the correct format (side-by-side) will work.

Roadmap

Currently SVTK is a useful and functional tool for exploring stereo imaging, and allows you to get going with your I3DR stereo camera quickly. There are a number of features/improvements in development including:

• Live camera calibration (currently in beta)
• Automated calibration correction
• Linux support
• ROS support
• Unit tests and other deployment improvements

See issues for details on known bugs and future enhancements

Stereo matching support

We have included support for two of OpenCV's matchers: the basic block matcher and semi-global block matching. The block matcher will run at over 60fps on a fast CPU (e.g. i5.) SGBM should provide better results, but will run around a factor of five slower.

I3DR's own 3D matching algorithm (I3DRSGM) is also built into this application by default however a license is required to be able to run it. Please contact [email protected] for requesting a license.

Installation

Download and install the windows installer for the latest release here.

See the User Guide for details on using the software as well as building from source.

License

This code is provided under the MIT license, which essentially means it's open source, but we require you to add our copyright if you distribute it elsewhere.

Previous Releases

See release for previous builds.

Developer zone

See User Guide for details on building from source.

Documentation can be found here.

This application is under active development and it is likely that in the short term there may be breaking changes to classes, or significant changes to the GUI.

Please read the guides for contributors and maintainers using this repository.

Please read the guide for branches and how they are used in this repository.

To clone the repository, make sure the clone with recurrsive to also include the sub-modules.

Using the follow command:

git clone https://github.com/i3drobotics/stereo-vision-toolkit.git --recursive

3rd Party Content

This repository is used for internal development and so we include both debug and release libraries and DLLs. This makes for quite a large repository for a rather small codebase (around 500MB), so be warned.

The project uses OpenCV for image processing, PCL and VTK for point cloud visualisation and hidapi for camera control. PCL requires Boost and Eigen which are included here. These dependencies are provided in accordance with their respective licenses which may be found in the license folder. We also use FontAwesome for icons via QtAwesome, along with QDarkStyle.

Automatic install

Some libraries have been moved to be externally downloaded to reduce the repostiory size. To download these libraries use the '3rdparty.bat' script provided in 'scripts'. This will download the libraries of the correct versions to the correct folders.
Note: This process builds boost with the required boost libraries so will take some time to complete.
Note: These scripts use command line features that are only avaiable in Windows 10 'curl' and 'tar'. If you are on a Windows 8 or lower machine you will need to install the Windows version of these commands and add them to your PATH variable.

cd PATH_TO_REPO/scripts
3rdparty.bat

Boost

Boost requires Visual Studio 2017, and needs cl.exe to be in the system PATH variable or to run '3rdparty.bat' in a Visual Studio command prompt.

QT

Tested with QT 5.14.2 with Visual Studio 2017.
Requires QT Webengine module for use in Fervor.

OpenCV Contrib & CUDA

There is limited usage of CUDA for certain image processing steps (e.g. rectification) and neural network support and OpenCV's ximageproc from contrib modules for WLS filter.

VCRedist

There is an issue where QT doesn't use the generic vc redist package and instead pulls from the system. Therefore the generic 2015-2019 vc redist is provided in this repository and can be updated using the script in '3rdparty/vcredist'.

Phobos control

Arduino code for controlling Phobos cameras is provided in src/camera/camera_control. This is for the serial communication between the arduino and this toolkit. See issue for more information.

stereo-vision-toolkit's Issues

Dialog box when loading a video to define if the video is pre-calibrated

from issue: #9
To avoid confusion a dialogue box should be added when a user loads a video to tell the program if it uses pre-calibrated images. This is to avoid double rectifying the video feed.

Stereo matcher doesn't work until single shot image has been taken (Deimos only)

After enabling the stereo matcher the disparity image doesn't appear unless single shot mode is toggled on and off. This only happens with the Deimos camera, Phobos works as expected.

Stereo matched enabled but not showing

Single shot acquisition is clicked. Disparity window is working.

Set back to free-running. Disparity window is working.

TIS camera can cause crash if left in unsafe state

If a TIS camera is manually removed from the computer while the program is running, this can leave the camera in an unsafe state. When the camera is connected the TIS library is unable to recognise the camera is in this state and continues as normal however is unable to capture images. The program does not immediately crash, however, if they try to disconnect the camera (using the GUI) then as the camera is not properly connected, this causes a crash.

CameraImagingSource::open returns with success as both 'handle.openDev()' and 'handle.isDevValid()' return with success even if the camera is in an unsafe state and unable to capture images.

cameraimaginsource.cpp

bool CameraImagingSource::open(std::string serial){
    qint64 qSerial = QString::fromStdString(serial).toInt();
    this->serial = qSerial;
    handle.closeDev();
    bool res = handle.openDev(qSerial);

    if (!handle.isDevValid()) {
      debugMessage("Couldn't open camera");
      QMessageBox alert;
      alert.setText(QString("Couldn't open camera with serial %1.").arg(qSerial));
      alert.exec();
    }else{
        this->width = handle.getVideoFormat().getSize().cx;
        this->height = handle.getVideoFormat().getSize().cy;

        debugMessage("Opened camera");
    }

    return res;
}

Slow download of installer from releases

Downloading of the installer from releases is throttled by GitHub. Should be hosted on I3DR's own website or better server to increase download speeds. Alternative a VPN can be used to trick GitHub into thinking you are downloading from the US and the installer is downloaded at full speed.

Add ROS yaml export

Would be useful to be able to export ROS yaml files directly, post-calibration.

Please add a folder destination for cal files

Only generate depth values where disparity is valid

To increase depth generation ignore disparity values that are invalid

Register if Phobos has been unplugged

If Phobos is unplugged while the program is running errors can occur. Currently unable to get the callback to trigger when the device is unplugged. Ideally the tab windows should be disabled if the unplugged event it triggered.
(This process is currently working for Deimos however this uses a much simpler process)

Slow image saving for high res images

disable rectify image when loading in videos

Videos save the rectified images. The rectified box is checked as default and hence buggers up the matching when a video is loaded.

Basler cameras can cause crash if unstable connection

Pylon has a fault which causes an unstable connection if it cannot read frames at the rate it is expecting. This can happen if the exposure is too high or a network is slow.

Incorrect matcher displayed on startup

I3DRSGM matcher is selected from the menu by default however the block matcher is actually what's being used.

Add an option to rename the filenames

Find Linux replacement for camera driver (dshow)

Adjust Phobos hardware trigger rate in the GUI

It would be good to set the frames per second from the GUI as opposed to from re-uploading the arduino code.

Create detailed user guide for the software

Currently, the README is used as the user documentation on this software. This should be changed to a clear and fully documented guide on how to use the software as the README should be a reference guide for building the project and not normally shown to the user.

Basler USB camera settings are not updating in cameras

Add ability to use two generic usb cameras

Add option for selecting a pair of generic USB cameras for use as a stereo setup. (e.g. using opencv videocapture)

"Saving to ..." folder path is cut off if too long

UI in the bottom right corner when displaying the folder path for saving the image is cut off if the path is too long. Should move to show the last folder a priority. The image below shows how a long folder is currently displayed:

The full file path was actually: "D:\Users\monke\OneDrive\Desktop\Deimos\deimos_cal_new_lens"
This should be shown on the UI like so: "..\Deimos\deimos_cal_new_lens".
It is suggested to use the last two folders to make it clear where the folder is (but a check would need to be done to make sure this isn't too long).

Add Titania support

Load in image files

Would be nice to be able to load in the saved files so a single image pair could be re-processed and the different algorithms tested on them.

Version has to be updated manually in multiple files before new release

A script should be created to automatically change the version number for all the required files before a release. The following files require manually updating on every release:

• Appcast.xml
• ReleaseNotes.html
• stereo_vision_toolkit.pro
• installer.iss

Add ROS to toolkit

If license file is not provided for I3DRSGM the program crashes on selection

Should the user attempt to select the I3DRSGM from without a license present in the application then the program will crash.

Move depth/disparity/colormap generating code to matcher class

Currently, calibration data is stored in AbstractStereoCamera class so 3D calculation have to be handled by this class. This should be moved out to instead reside in AbstractStereoMatcher class.

Load video not longer works

When a video is loaded only the first frame is loaded and the video won't play

Installer is blocked by some anti-virus programs

Error when running installer if Anti-Virus is running.

This issue has been shown to happen when using Avast Anti-Virus which CyberCapture halts the program from running until it has scanned it. However, once this scan process has completed the install continues as expected. The CyberCapture shield can be temporarily disabled from Avast Sheilds Control.

Camera serials must be explicitly listed config file

Cameras cannot be connected to the software unless their serials are list in camera_serials.ini
This file was created to make sure the correct left and right cameras are used when using Basler and imaging source cameras.
This should be adjusted so that the cameras are named 'Phobos_left_camera' and 'Phobos_right_camera' and identified that way rather than through the config file.
All Deimos cameras should be identified from the USB name 'see3cam_usb'.

Add ability to load yaml calibration files

Allow loading of calibration files from yaml files in the ROS standard format. This program is currently able to save to this format but not load them.

right.yaml

%YAML:1.0
---
image_width: 752
image_height: 480
camera_name: rightCamera
camera_matrix: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 7.1986915332569538e+02, 0., 3.8611705586057991e+02, 0.,
       7.1973065384792415e+02, 2.1707606908934204e+02, 0., 0., 1. ]
distortion_model: plumb_bob
distortion_coefficients: !!opencv-matrix
   rows: 1
   cols: 5
   dt: d
   data: [ 5.4692851489905617e-02, 1.7969294889773926e-01,
       -9.9490828017961830e-04, 1.1834695736334770e-03,
       -7.2919907900254755e-01 ]
rectification_matrix: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 9.9999984928828911e-01, 5.4890792755394192e-04,
       1.1112440071782371e-05, -5.4892172649057425e-04,
       9.9999902710837418e-01, 1.2823678269067832e-03,
       -1.0408527394353828e-05, -1.2823737334987230e-03,
       9.9999917770429692e-01 ]
projection_matrix: !!opencv-matrix
   rows: 3
   cols: 4
   dt: d
   data: [ 7.2113228421293900e+02, 0., 3.8768557739257813e+02,
       -4.3839646250020571e+01, 0., 7.2113228421293900e+02,
       2.2237854957580566e+02, 0., 0., 0., 1., 0. ]

left.yaml

%YAML:1.0
---
image_width: 752
image_height: 480
camera_name: leftCamera
camera_matrix: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 7.2231096300275101e+02, 0., 3.8132548128799044e+02, 0.,
       7.2253391457795385e+02, 2.2823806575700860e+02, 0., 0., 1. ]
distortion_model: plumb_bob
distortion_coefficients: !!opencv-matrix
   rows: 1
   cols: 5
   dt: d
   data: [ 6.7421150815912004e-02, 4.4342562686773099e-02,
       8.5550180331082828e-04, 6.0570740170956604e-04,
       -2.9100604915281014e-01 ]
rectification_matrix: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 9.9999647781534939e-01, -1.1221691911551628e-03,
       -2.4052220691582631e-03, 1.1190847030256004e-03,
       9.9999855029739793e-01, -1.2833754440023269e-03,
       2.4066587466855141e-03, 1.2806792764920626e-03,
       9.9999628392022910e-01 ]
projection_matrix: !!opencv-matrix
   rows: 3
   cols: 4
   dt: d
   data: [ 7.2113228421293900e+02, 0., 3.8387570190429688e+02, 0., 0.,
       7.2113228421293900e+02, 2.2237854957580566e+02, 0., 0., 0., 1.,
       0. ]

Remove auto load cameras at startup

User may not want to load the camera straight away so remove pop up windows at startup and replace with selection GUI panel once the system is loaded.

Change I3DRSGM settings internally (without tmp.param)

I3DRSGM currently changes settings by updating a temporary param file. This is due to a bug that stopped parameters from being set correctly.

Install update automatically

The current system requires the user to download the new release using a link in the fervour update window and install manually. Should instead automatically download this installer and run it on a button press.
Need to be able to close the running application and run the installer. Look at detached processes here).

vcredist has a different name on some systems

Inno setup looks for vcredist, this has been found to have different names on some systems:

• vcredist_x64.exe
• vc_redist.x64.exe

Inno setup should be adjusted to account for these different files.

Pylon library does not fully terminate on close

Message:
Pylon library initialization reference count is not zero, a call to PylonTerminate() is missing.

Option to flip camera input in x or y

App flip x and flip x option to the application settings tab

Program doesn't close properly (will crash rather than close in pressing exit)

Complete code documentation

All header files should contain Doxygen generators.

• abstractstereocamera.h
• abstractstereomatcher.h
• calibrateconfirmdialog.h
• calibratefromimagesdialog.h
• calibrationdialog.h
• cameradisplaywidget.h
• cameraimagingsource.h
• cameraopencv.h
• chessboard.h
• disparityviewer.h
• mainwindow.h
• matcheropencvblock.h
• matcheropencvsgbm.h
• matcherwidget.h
• matcherwidgetopencvblock.h
• matcherwidgetopencvsgbm.h
• paramfile.h
• qstereocamera.h
• stereocalibrate.h
• stereocameradeimos.h
• stereocamerafromvideo.h
• stereocameraopencv.h
• stereocameraphobos.h

Move stereo camera capture scripts into generic library (for use in other projects)

FPS label jumps between multiple values

FPS counter in bottom right jumps between multiple values. Probably better to use an average instead.

Network problems after saving images from Phobos

After saving images from Phobos the images save successfully but the camera stops capturing. This is likely due to network problems after the delay between network captures causing a timeout which the program is unable to recover from.

I3DRSGM Settings not changing

I3DRSGM Changes it's settings by updating tmp.param however if the application folder needs admin rights to update this file the settings do not get updated. A temporary fix is to run the application as admin (right-click -> run as admin). This is only necessary as the previous parameter setting method had other issues.

Update images in readme with improved GUI

Screenshot the updates to the GUI and replace the old images in the readme.

Add downsampling to calibration files

To avoid needing to re-calibrate on downsampled images add an option to downsample to calibration values so that one calibration can be used for any downsampling factor.

Add ability to specify AOI

Add GUI for specifying Area Of Interest for the matching to be applied to. This saves time by only matching the chosen part of the image whilst keeping the detail of a high-resolution image.

Software binning of input improve to improve 3D generation speed

calibration didn't load

Didn't load directly after loading calibration. Calibration loads fine if the program is re-started.

Program is very slow if pylon camera have network issues

Update CMakeLists to match qt pro file

CMakeLists is currently lagging behind after some updates were made to the qt .pro file. This should be updated and tested.

GUI runs slow if calibrate from image is done on large resolution images

Change min disparity slider in I3DRSGM to match negative disparity slider

Block matcher and SGBM have a toggle for negative or positive disparity for more fine control of the slider. Enable this feature in I3DRSGM as well