stuarta0 / blender-photogrammetry

Blender addon for structure from motion tools to allow dense point cloud reconstruction from Blender's camera tracker

blender-photogrammetry

A photogrammetry addon for Blender 2.79 and 2.80 that allows conversion and processing between a number of photogrammetry formats, in addition to providing dense reconstruction straight from Blender's camera tracker.

In the interests of remaining user friendly, this addon comes with all precompiled binaries required to process data.

Once enabled, the photogrammetry settings can be found in the Properties panel > Scene tab.

Supported Formats

Inputs:

• Blender's motion tracker: Allows reading of the tracker and reconstruction data from a tracked movie clip. Good for generating a dense point cloud of a tracked movie clip for reference, rendering or simulation.
• Bundler: Reads the bundler format consisting of a bundle.out and list.txt file.
• COLMAP: Reads a COLMAP model folder containing cameras, images and points3D files.
• ImageModeler: Reads an Autodesk ImageModeler .rzi file.
• Meshroom: Reads a Meshroom cameras.sfm file.
• VisualSfM: Reads the NVM file format.

Outputs:

• Blender: From the given input, create cameras and mesh with vertices representing the point cloud from the input.
• Bundler: Output a bundle.out, list.txt and associated images to use with other photogrammetry tools.
• PMVS: Output the bundler file format, then run PMVS2 dense reconstruction on the dataset, resulting in a .ply point cloud.
• COLMAP: Output to a COLMAP workspace with sparse model, then run COLMAP dense reconstruction, resulting in a .ply point cloud, poisson mesh and delaunay triangular mesh. Only available with CUDA GPUs on Windows
• VisualSfM: Output to an NVM file with associated images to use with other photogrammetry tools.

Note: Since inputs are outputs can be mixed and matched, this addon can be used as a convertor between different photogrammetry formats (with the added benefit of having Blender integration).

Usage

After processing a dense reconstruction in the format of choice, you can import the resulting .ply file back into Blender. Alternatively, use another tool like Meshlab to further process the data.

In the case where Blender's motion tracking data is processed, the resulting .ply mesh can be imported back into the tracking scene in Blender with the correct alignment to the original camera. When importing the .ply file, ensure you use +Z up, +Y forward.

Note: It's recommended you show the system console so you can track the progress of a reconstruction. Blender will become unresponsive whilst processing.

Caveats

Precompiled binaries are provided for Linux and Windows on x86_64 hardware only. If you're using a different platform or architecture, you won't be able to process most formats.

If you'd like more platform support (like Mac), please compile the supporting binaries (Bundler, PMVS, COLMAP) and send through a pull request so others may benefit.

If you run into issues on linux, you may need to install libjpeg62 and libgfortran3.

Roadmap

• Enable UI updates during processing if Blender 2.8 supports it (not feasible with Blender 2.79b)

Sources

Bundler

Project page and precompiled binaries: http://www.cs.cornell.edu/~snavely/bundler/

Source code: https://github.com/snavely/bundler_sfm

License

Bundler is distributed under the GNU General Public License. For information on commercial licensing, please contact the authors at the contact address below. If you use Bundler for a publication, please cite the following paper:

Noah Snavely, Steven M. Seitz, and Richard Szeliski. Photo Tourism: Exploring Photo Collections in 3D. SIGGRAPH Conf. Proc., 2006.

PMVS

Project page: https://www.di.ens.fr/pmvs/

Source code and precompiled binaries: https://github.com/pmoulon/CMVS-PMVS

License

In case you use this software, include an acknowledgement that PMVS2 was developed by Yasutaka Furukawa (University of Illinois at Urbana-Champaign, University of Washington) and Jean Ponce (University of Illinois at Urbana-Champaign, Ecole Normale Supérieure).

PMVS is distributed under the GNU General Public License.

For commercial licencing of the software, please contact Yasutaka Furukawa.

Thanks

Special thanks to Noah Snavely, the author of Bundler, for giving me some pointers on getting the right matrix representations in the bundle.out file. Also thanks to Sebastian König for help in testing the addon under Linux and with various datasets.

License

Blender Photogrammetry Addon Copyright (C) 2019 Stuart Attenborrow

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

blender-photogrammetry's Issues

Importing Bundler and list.txt

I have imported a pointcloud file. How do I also import the "Bundler.out" file and "list.txt" containing the images paths?

saving Colmap Project file

I've used the photogrammetry tool using blender motion track exporting to Colmap which worked perfectly and imported the pointcloud back into blender. however, I wish to refine the point cloud processing in colmap in order to get a better dense cloud but found that the process didnt leave a native Colmap project (.ini) file or a database (.db).
Is there a way to add a step to the process to produce these two files?
thanks

Help for a consulting

Hello, really great addon.
As I also wrote on youtube, I tried to use it but got some errors in Blender (v. 2.91 on Windows).
The point is that I really need it for my projects, I've been looking for something link that for a long time.
Sometimes I used PfTrack to do that but the workflow is much more difficult.
So I would like to agree with you also for a paid consultation in order to solve the issue.
Thank you a lot.
Stefano

Rotation matrices are being interpreted from cursor / origin

I have some basic Bundler data which consists of cameras in a straight line on X (east-west) axis, then going up on Y (vertical), rotating 90 degrees on Y to the right and then moving straight forward further along the X axis. The problem I see is that when heading shifts 90 degrees to the right (on the same vertical axis), the add-on is somehow interpreting that as a translation. The cameras are shown to rotation around the cursor / world origin instead of around their respective local axes.

I attempted to fix this problem by switching Blender's transform orientation to "gimbal" and transform pivot point to "individual origins", but those do not seem to have an effect on how the rotation matrix is processed. Here is a small sample of that rotation:

Translation values are Z,Y,X which correspond to axis north-south, altitude / vertical, and east-west. The rotation should be happening on the vertical axis (middle / Y).

1614.15662 0 0
0.99383223 0.00806891 -0.110600464
-0.0017907226 0.99838704 0.056746747
0.11087996 -0.056198686 0.99224365
-0.071071476 -0.11664623 2.469451

1614.15662 0 0
0.97913843 0.021447742 -0.20205948
-0.008532568 0.9978766 0.06457322
0.20301536 -0.061502036 0.9772423
-0.07967277 -0.12266238 2.4893844

1614.15662 0 0
0.9516065 0.027723696 -0.30606624
-0.0058047106 0.9973665 0.07229442
0.30726448 -0.06701922 0.9492613
-0.08839952 -0.13129313 2.523389

1614.15662 0 0
0.89873374 0.03140694 -0.43736857
-0.001721531 0.99767673 0.068104506
0.4384914 -0.06045488 0.8966997
-0.09061018 -0.1356472 2.5761647

1614.15662 0 0
0.82490796 0.03398874 -0.56424433
0.0020853977 0.9980009 0.063165985
0.5652633 -0.053282805 0.82318807
-0.07985141 -0.13775776 2.6344905

1614.15662 0 0
0.70870894 0.03639164 -0.7045617
0.013551228 0.99778235 0.0651679
0.7053708 -0.055732753 0.7066441
-0.055255346 -0.14196688 2.7103112

1614.15662 0 0
0.5863702 0.04462327 -0.8088132
0.018017095 0.9975161 0.068096206
0.8098429 -0.05450206 0.5841098
-0.020925561 -0.14822748 2.7860377

1614.15662 0 0
0.45274648 0.057762425 -0.88976634
0.009771731 0.997518 0.06972974
0.8915857 -0.04026445 0.4510583
0.029267171 -0.15076374 2.8668447

I am aware my matrices are not "standard". This is a horizontal orientation / right-handed custom scanner. Nonetheless, we should not be seeing any translation during camera import if the those last three coordinates are stable.

Add support for other formats

Upon release there have been a few requests for other format support:

• Meshroom
• openMVS
• COLMAP native format

Implementing these would also enable automated meshing, thus removing the Meshlab step if users wanted a textured mesh.

There's another project that supports importing these formats here:
https://github.com/SBCV/Blender-Addon-Photogrammetry-Importer

Then it'll be a matter of ensuring the binaries for each software are distributable across linux and windows. Alternatively, modify the plugin to allow external binaries for each software; configured in the user prefs.

"unkown location"

Hello Stuart,

many thanks for creating this addon.
I used for a couple of projects.
Now, suddenly it has stopped working.

I have tried using it with Colmap, PMVS, visualSFM .
I also tried using it across different versions of blender (2.8, 2.81, 2.90.1, 2.91 beta, 2.92 )

Specs :
OS : windows 10
Processor : intel i7 7700
Graphics card : 1070 mobile.

Can you help me please? Console output below.
Thanks,
Jayraj

This is what is output in the console window.

Modules Installed (blender_photogrammetry) from 'D:\Softwares\Blender_Addons\blender-photogrammetry-1.1-windows.zip' into 'C:\Users\J\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons'
Problem importing photogrammetry module "openmvs": Attributes "importer" and/or "exporter" must be defined in module
Problem importing photogrammetry module "pycache": Attributes "importer" and/or "exporter" must be defined in module
Warning: class PHOTOGRAMMETRY_PT_preferences contains a property which should be an annotation!
C:\Users\J\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\blender_photogrammetry_init_.py:184
assign as a type annotation: PHOTOGRAMMETRY_PT_preferences.collection_name
Warning: class PHOTOGRAMMETRY_PG_master contains a property which should be an annotation!
C:\Users\J\AppData\Roaming\Blender Foundation\Blender\2.92\scripts\addons\blender_photogrammetry_init_.py:184
assign as a type annotation: PHOTOGRAMMETRY_PG_master.input
assign as a type annotation: PHOTOGRAMMETRY_PG_master.output
assign as a type annotation: PHOTOGRAMMETRY_PG_master.in_blender
assign as a type annotation: PHOTOGRAMMETRY_PG_master.in_bundler
assign as a type annotation: PHOTOGRAMMETRY_PG_master.in_colmap
assign as a type annotation: PHOTOGRAMMETRY_PG_master.in_imagemodeler
assign as a type annotation: PHOTOGRAMMETRY_PG_master.in_meshroom
assign as a type annotation: PHOTOGRAMMETRY_PG_master.in_visualsfm
assign as a type annotation: PHOTOGRAMMETRY_PG_master.out_blender
assign as a type annotation: PHOTOGRAMMETRY_PG_master.out_bundler
assign as a type annotation: PHOTOGRAMMETRY_PG_master.out_colmap
assign as a type annotation: PHOTOGRAMMETRY_PG_master.out_pmvs
assign as a type annotation: PHOTOGRAMMETRY_PG_master.out_visualsfm
[blender-photogrammetry] Platform: windows
process photogrammetry
in_blender
out_colmap

Could COLMAP-CL potentially be added in a future version?

Some people have made a version of COLMAP that runs on OpenCL instead of CUDA, called COLMAP-CL. This enables COLMAP to run on AMD GPUs as well as NVIDIA GPUs.

I did a quick test where I took this add-on, swapped out the original COLMAP exe and library folder with the library folder and exe that comes with the latest release of COLMAP-CL and it seems to work just fine on my AMD GPU, so hopefully little or minimal work would be required to add it.

Add debugging for bundle points

Other software has the ability to export debugging SVGs with image points on them (e.g. openMVG). Implement a simple SVG debug output like those tools:

<svg width="1920" height="1080" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
    <image xlink:href="/path/to/image/0001.jpg"/>
    <circle cx="1696.62" cy="728.95" r="5" fill="none" stroke="#f00" stroke-width=".5"/>
    <!-- repeat circles for as many points are visible for this camera -->
</svg>

Coordinates are (0,0) top left. If we set the SVG resolution the same as the bundler images, the point translation should be something like (x + width/2, height/2 - y)

Blender 2.9 version planned ?

Blender 2.9 version

Any blender 2.9 version of the add-on planned ?

ImageModeler rzi import issue.

Using version 1.1-linux.
Occasionally some files can't be imported, while others work perfectly fine.

Python: Traceback (most recent call last):
  File "/home/colby/.config/blender/2.92/scripts/addons/blender_photogrammetry/__init__.py", line 76, in execute
    data = inputs[p.input].func(extract_props, scene=scene)
  File "/home/colby/.config/blender/2.92/scripts/addons/blender_photogrammetry/imagemodeler/extract.py", line 82, in extract
    f = (float(cinf['fbw']) / 2) / tan(radians(float(intrinsics['fovx']) / 2))
KeyError: 'fbw'

location: <unknown location>:-1

Attached is one of the problematic files.
passat rzi.zip

Add masking

PMVS2 supports masks and so does Blender. We should add masking to the PMVS output.

From the PMVS documentation:

Segmentation masks can be optionally given to the program. A segmentation mask must be given in pgm file and stored under root/masks/ directory. Their names must be again 00000000.pgm, 00000001.pgm, ... A pgm file must be in P5 (binary grey-scale) format and there should be no comments. If the intensity of a pixel is less than 127, the pixel is treated as background, and if the intensity is more than 127, the pixel is treaded as foreground. Intuitively, background should appear black and foreground should appear white. If files do not exist in the directory, the program just ignores them. You do not have to place pgm files for all the input images. You can, for example, put only 0000001.pgm and 00000002.pgm to specify mask images for these 2 images.

Mac version?

Is there going to be Mac version any time soon? Or how to compile it for Mac?

Blender Output Meshroom

In some cases, for some kind of surfaces, a manual tracking is required:
Is there a possibility to export blender motion tracking data to Meshroom?

Thank you,
Riccardo

Meshroom node request

Hi @stuarta0 would you be interested in wrapping up your blender-photogrammetry plugin as a Meshroom node that can be used for format conversion within Meshroom? https://meshroom-manual.readthedocs.io/en/latest/feature-documentation/core/nodes.html#api It would be awesome to be able to utilize and compare different photogrammetry tools within Meshroom (as discussed here alicevision/Meshroom#1395)
I would not expect you to work for free, so if some financial contribution would help, you can contact me via [myGHusername]@posteo.de

Camera world matrix should calculate from start of valid bundle

Code in export_bundler.py line 82 sets frame to scene.frame_start before calculating the camera matrix_world. In testing, it needs to be the frame where the movieclip tracking starts without error.

The following records the earliest any marker appears, but it's not the point where the tracking becomes valid.

frame_min = 99999
for t in clip.tracking.tracks:
    frame_min = min(frame_min, t.markers.values()[0].frame)

Add-on doesn't work (Windows 10)

Hello,

Tried your add-on, but it seems that something is wrong. Not sure what's the problem, because I have COLMAP installed and it works actually. I've tried 2.79b, 2.8 versions but they both doesn't work. PMVS doesn't work too. My GPU is NVIDIA NVS 4200M.

Here is the log:

process photogrammetry
in_blender
out_colmap
ERROR (bke.library): c:\b\win64_cmake_vs2017\win64_cmake_vs2017\blender.git\source\blender\blenkernel\intern\library.c:191 id_us_ensure_real: ID user count error: SCScene (from '[Main]')
Saved: 'C:\Denis\moveai\4th_september\photogrammetry_example\renderoutput\photogrammetry\0004.jpg'
Time: 00:01.77 (Saving: 00:00.43)

Saved: 'C:\Denis\moveai\4th_september\photogrammetry_example\renderoutput\photogrammetry\0006.jpg'
Time: 00:01.60 (Saving: 00:00.29)

Saved: 'C:\Denis\moveai\4th_september\photogrammetry_example\renderoutput\photogrammetry\0008.jpg'
Time: 00:01.65 (Saving: 00:00.28)

Saved: 'C:\Denis\moveai\4th_september\photogrammetry_example\renderoutput\photogrammetry\0010.jpg'
Time: 00:01.70 (Saving: 00:00.28)

{'cameras': {}, 'resolution': (1920, 1080), 'trackers': {}}
[ReadBundleFile] Bundle version: 0.300
[ReadBundleFile] Reading 0 images and 0 points...

@@ Conversion complete, execute "sh pmvs/prep_pmvs.sh" to finalize
@@ (you will first need to edit prep_pmvs.sh to specify your bundler path,
@@ so that the script knows where to find your
@@ RadialUndistort and Bundle2Vis binaries)
[ReadBundleFile] Bundle version: 0.300
[ReadBundleFile] Reading 0 images and 0 points...
[WriteBundleFile] Writing 0 images and 0 points...
C:\Users\Mania\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\blender_photogrammetry\colmap\bin\colmap.exe patch_match_stereo --workspace_path C:\Denis\moveai\4th_september\photogrammetry_example\colmap\pmvs --workspace_format PMVS --pmvs_option_name reconstruction
Reading workspace...
Importing PMVS workspace (option reconstruction)...
F0820 00:25:24.831068 14032 workspace.cc:188] Check failed: range_lower < range_upper (0 vs. 0)
*** Check failure stack trace: ***
Traceback (most recent call last):
File "C:\Users\Mania\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\blender_photogrammetry_init_.py", line 79, in execute
outputs[p.output].func(load_props, data, scene=scene)
File "C:\Users\Mania\AppData\Roaming\Blender Foundation\Blender\2.80\scripts\addons\blender_photogrammetry\colmap\load.py", line 48, in load
raise Exception('COLMAP patch_match_stereo failed, see system console for details')
Exception: COLMAP patch_match_stereo failed, see system console for details

location: :-1

location: :-1

It would be nice if you can help. Thanks!