alicevision / alicevision

Photogrammetric Computer Vision Framework

AliceVision is a Photogrammetric Computer Vision Framework which provides a 3D Reconstruction and Camera Tracking algorithms. AliceVision aims to provide strong software basis with state-of-the-art computer vision algorithms that can be tested, analyzed and reused. The project is a result of collaboration between academia and industry to provide cutting-edge algorithms with the robustness and the quality required for production usage.

Learn more details about the pipeline and tools based on it on AliceVision website.

See results of the pipeline on sketchfab.

Photogrammetry

Photogrammetry is the science of making measurements from photographs. It infers the geometry of a scene from a set of unordered photographies or videos. Photography is the projection of a 3D scene onto a 2D plane, losing depth information. The goal of photogrammetry is to reverse this process.

See the presentation of the pipeline steps.

License

The project is released under MPLv2, see COPYING.md.

Citation

If you use this project for a publication, please cite the paper:

@inproceedings{alicevision2021,
  title={{A}liceVision {M}eshroom: An open-source {3D} reconstruction pipeline},
  author={Carsten Griwodz and Simone Gasparini and Lilian Calvet and Pierre Gurdjos and Fabien Castan and Benoit Maujean and Gregoire De Lillo and Yann Lanthony},
  booktitle={Proceedings of the 12th ACM Multimedia Systems Conference - {MMSys '21}},
  doi = {10.1145/3458305.3478443},
  publisher = {ACM Press},
  year = {2021}
}

Bibliography

See Bibliography for the list of research papers and tools used in this project.

Get the project

Get the source code: git clone --recursive git://github.com/alicevision/AliceVision

See INSTALL.md to build the project.

Continuous integration status: .

Launch 3D reconstructions

Use Meshroom to launch the AliceVision pipeline.

• Meshroom provides a User Interface to create 3D reconstructions.
• Meshroom provides a command line to launch all the steps of the pipeline.
• Meshroom is written in python and can be used to create your own python scripts to customize the pipeline or create custom automation.

The User Interface of Meshroom relies on Qt and PySide. The Meshroom engine and command line has no dependency to Qt.

Contact

Use the public mailing-list to ask questions or request features. It is also a good place for informal discussions like sharing results, interesting related technologies or publications:

[email protected] http://groups.google.com/group/alicevision

You can also contact the core team privately on: [email protected].

Contributing

Beyond open source interest to foster developments, open source is a way of life. The project has started as a collaborative project and aims to continue. We love to exchange ideas, improve ourselves while making improvements for other people and discover new collaboration opportunities to expand everybody’s horizon. Contributions are welcome. We integrate all contributions as soon as it is useful for someone, don't create troubles for others and the code quality is good enough for maintainance.

Please have a look at the project code of conduct to provide a friendly, motivating and welcoming environment for all. Please have a look at the project contributing guide to provide an efficient workflow that minimize waste of time for contributors and maintainers as well as maximizing the project quality and efficiency.

Use github Pull Requests to submit contributions:

http://github.com/alicevision/AliceVision/issues

Use the public mailing-list to ask questions or request features and use github issues to report bugs:

http://github.com/alicevision/AliceVision/pulls

Project history

In 2009, CMP research team from CTU started the PhD thesis of Michal Jancosek supervised by Tomas Pajdla. They released windows binaries of their MVS pipeline, called CMPMVS, in 2012. In 2009, Toulouse INP, INRIA and Duran Duboi started a French ANR project to create a model based Camera Tracking solution based on natural features and a new marker design called CCTag. In 2010, Mikros Image and IMAGINE research team (a joint research group between Ecole des Ponts ParisTech and Centre Scientifique et Technique du Batiment) started a partnership around Pierre Moulon’s thesis, supervised by Renaud Marlet and Pascal Monasse on the academic side and Benoit Maujean on the industrial side. In 2013, they released an open source SfM pipeline, called openMVG (“Multiple View Geometry”), to provide the basis of a better solution for the creation of visual effects matte-paintings. In 2015, Simula, Toulouse INP and Mikros Image joined their efforts in the EU project POPART to create a Previz system based on AliceVision. In 2017, CTU join the team in the EU project LADIO to create a central hub with structured access to all data generated on set based on AliceVision.

See CONTRIBUTORS.md for the full list of contributors. We hope to see you in this list soon!

alicevision's Issues

openMVG2CMPMVS remap option

Add a remap path option (linux/windows conversion).

[calibration] Add checkerboard calibration application

• based on OpenCV
• choose a file format for the calibration result

ComputeMatches does not save the matches.f.txt

The file is created but it's always empty. There must be a bug of some sort in ExportPairedIndMatchFile
The other function ExportPairedIndMatchFilePerImage works, instead.

Compression of feat/desc files

Zip provide a 1.5 compression ratio.

Fedora/OpenSuse crash with cereal

It seems that files generated on one system are not loadable by the same binaries on another system.

[calibration] Camera calibration format

Decide the format to use for the camera calibration file. Note that we must be able to deal with:

• a camera with fixed intrinsics parameters
• a camera with intrinsics parameters changing over time (f, pp, and k3)
• a camera rig composed of N (N = {1,2,3}) cameras

So far for the camera localization (voctree or cctag based) a simple txt file is used using a format like this

// the structure of the file is
// int #image width
// int #image height
// double #focal
// double #ppx principal point x-coord
// double #ppy principal point y-coord
// double #k0
// double #k1
// double #k2

as explained here https://github.com/poparteu/openMVG/blob/voctree_localizer/src/openMVG/dataio/IFeed.cpp#L18

Save intermediate results of Incremental pipeline in Alembic

Need #2

Add documentation to the global reconstruction method

Add bounds for principal point and focal in BA

To avoid degenerate cases

Color as an option of file format conversion

• Merge ComputeSfM_DataColor into an option of ConvertSfM_DataFormat.
• export point colors in ABC

Grid in openCV feature detection

Compilation problem for cctag + boost 1.58

When compiling with cctag enabled and boost version 1.58 a compilation error arises due to the definition of the macro B2() in metric_hamming.cpp of openmvg,

static const unsigned char pop_count_LUT[256] =
{
#   define B2(n) n,     n+1,     n+1,     n+2
#   define B4(n) B2(n), B2(n+1), B2(n+1), B2(n+2)
#   define B6(n) B4(n), B4(n+1), B4(n+1), B4(n+2)
    B6(0), B6(1), B6(1), B6(2)
};

which happens to be the name of a variable in one of the header of boost (boost/variant/variant.hpp:194), which is used only by cctag. The simplest workaround is to undef the macro just after use (it is not used elsewhere). Maybe there is another better way to do so

Add CCTag support

• add cctag describer
  • in ComputeFeatures
• create localizer based on cctag
• add proper cmake options to compile with cctag in omvg

Incremental: Fix bundle adjustment to avoid drift

• The result of the reconstruction is always in a random position and orientation. We would like to keep it centered in 0.
• Fix the UP: currently the reconstruction are always flipped vertically
• Sometimes the reconstruction is discretised in a grid. It seems to come from floating point precision at the reconstruction scale. So one solution could be to add a constraint in the bundle to work in [0,1] value range. (see also openMVG/openMVG#471)

[Parameters] Two different sets of parameters for natural or cctag features.

Create a class of parameters used both in the SfM and the camera localizsation:
This class should hold all the thresholds (usually hardcoded) in all the function.

This set of parameters MUST BE different depending whether natural or artificial features are used. This mainly comes from the fact that the artificial features ensure a better accuracy compared to the natural ones but are much less sparse in the reconstruction.

Alembic exporter

Need #14
Todo:

• Create an exporter class inside the library
• Visibility list
• Index for views, poses, intrinsics
• Use real sensor width (if exists)

Threshold on matching

Keep metadata on Views

Test GT Levallois dataset

• Script to set custom attributes in Maya
• Export ABC with reconstructed cameras
• Export ABC with virtual camera motion
• Add sfmdata in input of evalQuality (--gt)
  https://github.com/poparteu/openMVG/blob/alembic_io/src/software/SfM/main_evalQuality.cpp#L88
  "--gt": Can be a folder (as currently) or a standard sfmdata file
• Support missing cameras
  https://github.com/poparteu/openMVG/blob/alembic_io/src/software/SfM/main_evalQuality.cpp#L129
  We print the number of reconstructed cameras and compare the reconstructed ones (based on their ID).
• Create test for reconstruction
• Create test for camera localization
• Create a dedicated github repository with the data

[travis] cache Ceres

Make travis compile the repository using a cached copy of Ceres. It will speed up a lot (50%) the compilation time

Add OpenCV in Travis

ABC export finishing

• use the correct sensor size if available
• export visibility
• create a global transform and a transform for all cameras to have a nice hierarchy
• add prefix for all custom properties (and maybe better names)
• name pointCloud as "mvgPointCloud"
• check compatibility export/import

In the continuity of #2

Find correct parameters for onset usage

Like ~100 images in less than 15 minutes for SfM without losing too many cameras

Linked to http://github.com/poparteu/openMVG/issues/22

Migrating from legacy to container-based infrastructure for Travis

This migration is needed for two main reasons :

• have 2 cores instead of 1,5
• simplify the installation of some dependencies (especially boost-1.53)

[localization] Resectioning for camera RIG externally not calibrated

Resectioning from camera RIG externally not calibrated:

• N unknown extrinsics + M known extrinsics (N+M=3)
• every camera does the localization
• BA with rig constraint to refine everything

This is should be (relatively) easy once #12 is done

Tests with ground truth on travis

• add the ground truth test
• change to incremental pipeline
  • how to choose the A/B pair ? -> automatic
• perform quality evaluation (ratio ? time ?)
  • caymard/SfM_quality_evaluation#1
• reduce time by using travis cache

[calibration] Calibrate a RIG of cameras from an existing reconstruction using CCTAG

Perform camera localization of multiple videos from an externally uncalibrated RIG and use this rigid contraint to calibrate the RIG. Each of the camera is internally (intrinsics) calibrated

• Compute camera poses of N images sequences (depends on #17)
• Compute the rigid relative poses
• Choose the best relative pose that minimize the reprojection error in all images
• Bundle adjustment of all camera poses using the rigid relative pose constraint

Next is test results wrt GT #68

Evaluation of SIFT descriptors: uchar vs float

add unity tests for kmeans

• port the unity tests

Resectioning with auto-calibration

Estimate intrinsics directly in the resectioning.

Currently, we need an initial guess of the focal length and we refine all instrinsics in the Bundle Adjustment.

[sfm] Use simultaneously sift and cctag features.

Be able to use both natural regions and CCTag regions in the entire SfM pipeline.

• create a class SIFT_CCTAG_Image_describer that transparently uses the two feature extractors [3a55b1e]
• when matching features use brute force or FLANN rather than the default method
• it would be nice to have #50 as CCTag have high rejection rate

Alembic importer

For some internal tests, we need to be able to import alembic cameras in the sfmdata format.
Todo :

• point cloud (structure)
• visibility list
• focal conversion metric/pixel
• index for views, poses, intrinsics
• fusion common intrinsics
• handle file opening problem

[localization] Partially calibrated Camera RIG localization in an existing reconstruction

Resectioning from externally calibrated camera RIG: 2 calibrated cameras + 1 camera with unknown intrinsics (the 3 are externally calibrated, though):

• this is useful for the main camera (shooting camera) as it may change the focal length during the shooting while the witness are supposed to stay fixed)
• RIG camera pose estimation using the N calibrated cameras
• for each uncalibrated camera
  • use normal localization (voctree+matching)
  • use 3D points and pose of the main camera (which is known from the extrinsics) ==> this in case of large occlusions on the main camera preventing the use of voctree
  • BA to refine intrinsics of the main camera

depends on #13

[tests] setup to compare results with different parameters

Add vocabulary tree

• add the main classes
• reimplement/port readFeatures, possibly using automatic switching desc/desc.zip
  • careful with loading char/float
  • decide what to load: so far it expects eigen float features as output object
  • it should work with a simple list.txt with the name of the .desc to load as well as a sfm_data to create the voctree from a openmvg dataset
• port the essential apps
  • createvoctree
  • pairlistFromVocTree
  • testvoctree
• code style: spaces et al
• CAREFULLY update/merge with a more recent develop (@fabiencastan)

[voctree] add new distance methods

The goal is to improve the results for partial matches.

[voctree] SIFT descriptors in float

If we have SIFT descriptors in char in input we have to convert them in float for the voctree.

[localization] Calibrated Camera RIG localization in an existing reconstruction using CCTag

Resectioning from externally calibrated camera RIG: all cameras are calibrated both internally (intrinsics) and externally (extrinsics):

• We collect 2d-3D association from each camera,
• apply the camera subpose to the 3D points of each camera
  • ie we collect the 2d-3d correspondences for the camera 1 (which has pose1 wrt camera 0)
  • we apply the pose1 transformation to those 3D points so
• do a global resectioning with all the 3D-2D associations from all the cameras of the rig
  • we need to undistort all the 2D points and put them in normalized coordinate system (ie inv(K)*feat) as they must be independent from their camera parameters (we are considering the associations as if they were from the same camera)

Do the same for SIFT #69

CompteFeatures/Matches parallelization

Add option to specify index ranges to compute a only specific part. This allows to compute in parallel on render farm.

[alembic] export animated cameras

Currently cameras are exported individually.
A nice feature would be to have rather an animated camera. This is better if for example we want to export the movement of the camera rig, so we have just 3 cameras moving around and not 3 series of cameras
it goes with #2 and #34

Use lensfun to provide an initial intrinsic guess

[cmake] write a proper FindAlembic

able to

• deal with all-in-one library and the older version with many different libs
• deal with an input option to select static vs shared

Once done,

• PR it to the official repository

Camera Localization inside an existing reconstruction

Close to the incremental method, but do not modify the 3D reconstruction (cameras & 3D points).

• low level api for
  • matching
  • resection
• bundle adjustment for a single camera after resection
• port the class from cameraLocalization

[matches] don't load all the descriptors if we have a "pair_list" input

See here:
https://github.com/poparteu/openMVG/blob/dev_mikros/src/software/SfM/main_ComputeMatches.cpp#L242

[calibration] Calibrate a RIG of cameras from an existing reconstruction using natural features (or SIFT_CCTAG)

same as #12 but with SIFT and SIFT_CCTAG
For this we just need to adapt #12 so that we can switch from cctag_localizer to voctree_localizer.

• Create a common Interface ILocalizer between cctag_localizer and voctree_localizer
  • move the init() method to constructor of each class
  • create a common base for the parameters
  • modify localize() method to use parameters from the proper class
• add a flag for the command line script to chose which mode to use

Add new images in an existing reconstruction

Restart the incrementalEngine from an existing reconstruction.

Add an option in the imageListing to generate a UID for the cameras

[voctree] fix sanityCheck

[localization] Use simultaneously sift and cctag features

• modify voctree localizer so that it can optionally load CCTag_SIFT features
  • the feature extractor used internally should be initialized dynamically according to the loaded features
• add an option on the main script
• the localization procedure should be unaffected by the type of feature used

Merge existing reconstructions