isl-org / open3d Goto Github PK
View Code? Open in Web Editor NEWOpen3D: A Modern Library for 3D Data Processing
Home Page: http://www.open3d.org
License: Other
Open3D: A Modern Library for 3D Data Processing
Home Page: http://www.open3d.org
License: Other
During the process of implementing OdometryRGBD, we found a need of basic image processing modules (e.g., pixel pointer, downsampling, filtering, creating pyramid image) that are generally useful for OdometryRGBD, texturing meshes, and so on.
Plans:
Jaesik
When using g++-6 on OSX (for the sake of OpenMP), there are compilation errors linking Open3D against brew-installed jsoncpp. After digging it for a while, it is due to conflict between libc++ (used by llvm by default) and libstdc++ (used by g++ by default).
See this post for details:
open-source-parsers/jsoncpp#597
The current workaround is 6eef4a9
We force a compilation of jsoncpp from source when g++ compiler is used on OSX.
This however is a temporary solution. In the long run, we should switch to:
http://libcxx.llvm.org/docs/UsingLibcxx.html#using-libc-with-gcc
Link everything to libc++ on OSX.
This issue is related to #1
I have discussed with Jaesik.
We need to demonstrate that we have the state-of-the-art registration results.
It also helps us debugging the reconstruction system.
OpenMP conveniently parallelize many algorithms and accelerates them greatly. However, the default Xcode LLVM does not support OpenMP.
The current solution is to use homebrew-gcc on OSX, see
https://github.com/IntelVCL/Open3D/blob/master/docs/compilation.rst
It however has some compatibility issues with some external libraries (glfw). It also introduces conflicts between libc++ and libstdc++, see #8
We have experimented a homebrew-llvm. It is supposed to support OpenMP. However, it is hard to configure it right. This should be resolved in the future.
Looks like the current image IO can handle jpg and png image format. However, some of RGBD dataset (such as NYU dataset) is captured with ppm/pgm format. Do we need to extend OpenCV support for these formats as well?
This module is built based on the odometry and registration module.
I have been wanting to it for a long time, but have not got time for it.
The idea is to use the lambda function hooks to play customized animation in Python using GUI. This will solve lots of issues.
E.g. we want to access raw data (in numpy format) of a screen capture stream in an animation, etc.
As title.
This function has for-loop with OpenMP parallelization. However, it actually uses single core on run time. This is A bottleneck for RegistrationRANSACBasedOnFeatureMatching
. I found this by using Visual Studio Profiler.
Let me put code snippet for a reference:
#ifdef _OPENMP
#pragma omp parallel for schedule(static)
#endif
for (int i = 0; i < (int)source.points_.size(); i++) {
std::vector<int> indices(1);
std::vector<double> dists(1);
const auto &point = source.points_[i];
if (target_kdtree.SearchHybrid(point, max_correspondence_distance, 1,
indices, dists) > 0) {
#ifdef _OPENMP
#pragma omp critical
#endif
{
result.correspondence_set_.push_back(
Eigen::Vector2i(i, indices[0]));
error2 += dists[0];
}
}
}
I think something is happening to target_kdtree.SearchHybrid
. Here are a few suspicious things:
target_kdtree.SearchKNN
, the app crashes on run-time.Is the code is reliable? Is the issue related to low level KDTree operations?
This issue is derived from #24
This issues is derived from #24. Lower priority.
Need not to be done before the first launch.
Travis CI detects some compile errors in Ubuntu trusty.
:
[ 19%] Building CXX object IO/CMakeFiles/IO.dir/ClassIO/PinholeCameraTrajectoryIO.cpp.o
[ 20%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileBIN.cpp.o
[ 21%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileJPG.cpp.o
[ 21%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FilePLY.cpp.o
[ 22%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileXYZN.cpp.o
[ 23%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileLOG.cpp.o
[ 24%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileXYZRGB.cpp.o
[ 25%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FilePCD.cpp.o
[ 25%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FilePNG.cpp.o
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:36:47: error: ‘png_image’ has not been declared
void SetPNGImageFromImage(const Image &image, png_image &pngimage)
^
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp: In function ‘void three::{anonymous}::SetPNGImageFromImage(const three::Image&, int&)’:
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:38:11: error: request for member ‘width’ in ‘pngimage’, which is of non-class type ‘int’
pngimage.width = image.width_;
^
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:39:11: error: request for member ‘height’ in ‘pngimage’, which is of non-class type ‘int’
pngimage.height = image.height_;
^
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:40:11: error: request for member ‘format’ in ‘pngimage’, which is of non-class type ‘int’
:
This is because png.h
is not correctly included by Linux cmake build chain.
I looked /src/External/CMakeLists.txt
. If cmake detects libpng
, PNG_INCLUDE_DIRS
in /IO/CMakelists.txt
is weirdly set to as /usr/include/usr/include
. If libpng
is built from source, the correct header is included and this issue is gone.
I was also regenerate the same error using my Linux workstation with CMake 3.8.2
So far the best file dialog library is tinyfiledialogs: https://sourceforge.net/projects/tinyfiledialogs/
It is small, clean, cross-platform. It has all functionalities I need.
With it, we shall have Ctrl/Cmd+O/S, ...
Roadblock: I need to do it.
Problems: On Windows, the input box is ugly (I checked, there seem to be no better way. Windows does not have native input box support). On Windows, the file boxes are slightly blurred on 4K monitors (there seem to be no easy workarounds).
Other options (I considered and decided against them):
Native File Dialog
https://github.com/mlabbe/nativefiledialog
It does not have message box etc.
Qt/GTK3+
Too complicated to use/include in External.
I was trying to add timer to my graph optimization loop. I want to just retrieve the time duration.
For example, I would like to add time : xx.xxx sec.
to the last in the following line.
PrintDebug("[Iteration %02d] residual : %e, lambda : %e, valid edges : %d/%d\n, time : %2.3f",
iter, current_residual, current_lambda,
valid_edges, n_edges - (n_nodes - 1), /* get time duration here */);
I tried to use GetDuration();
in Core\Utility\Timer.h
but it is protected function. Can I just get time of duration from Timer
? This would be useful if I want to customize timing messages.
There is an compilation failure from Travis CI. Related to #92.
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp:36:47: error: ‘png_image’ has not been declared
void SetPNGImageFromImage(const Image &image, png_image &pngimage)
^
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp: In function ‘void three::{anonymous}::SetPNGImageFromImage(const three::Image&, int&)’:
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp:38:11: error: request for member ‘width’ in ‘pngimage’, which is of non-class type ‘int’
pngimage.width = image.width_;
^
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp:39:11: error: request for member ‘height’ in ‘pngimage’, which is of non-class type ‘int’
pngimage.height = image.height_;
We have Travis CI for Unix systems but we also need to CI for Windows.
One famous tool is:
https://ci.appveyor.com/
Per request by Arno.
Using Visualizer in Python is still quite tedious.
This should be fixed.
def custom_draw_geometry_with_rotation(pcd):
def rotate_view(vis):
ctr = vis.GetViewControl()
ctr.Rotate(10.0, 0.0)
return False
vis = Visualizer()
vis.CreateWindow()
vis.AddGeometry(pcd)
vis.RegisterAnimationCallback(rotate_view)
vis.Run()
vis.DestroyWindow()
#33 only handles Ubuntu trusty (14.04). We need more. Some candidates below.
Open3D has PinholeCameraIntrinsic
and PinholeCameraTrajectory
that are good to read camera information from file. However, we don't have a file IO support for dense matching point cloud pairs. This is essential to GraphOptimization
module.
Bug submitted by Arno.
As title says, point cloud registration module does not outputs information matrix. Information matrix is necessary for global graph optimization.
I am not trying to output information matrix for every case as computing information matrix would add additional computational burden, and not used for normal cases. I am going to introduce additional option for registration class, and registration result class will include information matrix if it is specified. How do you think?
I notice that cmake 3.6 tool-chain generates py3d
instead of py3d.so
after compilation.
This affects python scripts. They don't recognize py3d
library.
For example:
> python
>>> from py3d import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named py3d
But if I change the name, it works
> mv py3d py3d.so
> python
>>> from py3d import *
(no error)
This issue comes with MacOSX Sierra, Cmake 3.6, Mac Native Python 2.7.
Windows + VisualStudio2015 seems to fine.
We may need to change CMakeList.txt file or get some information from Pybind to fix this issue.
ConvertFromPinholeCameraParameters
function in Open3D/src/Visualization/Visualizer/ViewControl.cpp is designed for deterministic pinhole camera model: its principal point should be central position of an image domain.
For example, ViewGeometry terminates if below condition does not meet:
intrinsic.intrinsic_matrix_(0, 2) != (double)window_width_ / 2.0 - 0.5 ||
intrinsic.intrinsic_matrix_(1, 2) != (double)window_height_ / 2.0 - 0.5)
Can we extend current OpenGL interface to allow principle point of non-centered position?
On Windows, all packages are included in External, and linked with Open3D.
On Ubuntu/OSX, this should be done automatically with native libraries.
This should be done with findpackage in CMake.
Roadblock: Need to do it and thoroughly test if these dependencies (a weird version, e.g.) break Open3D. In that case, we should fall back to the code included in External.
I encountered compile error with up-to-date master branch. It is due to libpng.
Cmake says there is libpng in my Mac, but build tool-chain is bit broken somehow. The tool-chain cannot link libpng to the objects.
-- Checking for one of the modules 'libjpeg'
-- Build libjpeg from source
-- Use native build of libpng
-- Use native build of glfw
:
[ 74%] Linking CXX executable ../bin/Test/TestFileDialog
ld: library not found for -lpng16
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I am using -DOpen3D_USE_NATIVE_DEPENDENCY_BUILD=OFF
for now to avoid this failure case.
We have fancy class PoseGraph
and its derived class: PoseGraphNode
and PoseGraphEdge
.
A question arises If I want to make PoseGraph
and grow their members in python. For example, I get odometry and want to save the trajectories into .json
file. I would need to use PoseGraph
as it is intended for this case.
So. I want to add new variable. the hope is:
# in python
pose_graph = PoseGraph()
pose_graph.nodes.append(PoseGraphNode(T))
pose_graph.edges.append(PoseGraphEdge(s,t,T,...))
what I am not clear is that whether we can use .append
or std::vector<> push_back()
operator in python.
In our c++ implementation the member value nodes_
and edges_
in PoseGraph
are in std::vector
format so I think this would be possible. Is there natural way to append new items to PoseGraph
?
In python
pcd_1 = pcd_0
does not hard copy. Instead, pcd_1
gets the pointer of pcd_0
. I think It is not intuitive for python. Maybe need to define =
operators for basic data types?
This issue is derived from #41
I had benchmarked several ways for building Jacobian matrix.
Please check the code in Option 1 and Option 2.
// data generation
const int NCORR = 100000000;
time_t tstart, tend;
std::vector<Eigen::Vector3d> data;
data.resize(NCORR);
for (size_t i = 0; i < NCORR; i++) {
data[i] = Eigen::Vector3d::Random();
}
// Option #1
// Directly Building JtJ and Jtr
tstart = time(0);
Eigen::Matrix6d ATA;
Eigen::Vector6d ATb;
Eigen::Vector6d A_r;
double r;
ATA.setZero();
ATb.setZero();
for (size_t i = 0; i < NCORR; i++) {
const Eigen::Vector3d &vs = data[i];
const Eigen::Vector3d &vt = data[i];
const Eigen::Vector3d &nt = data[i];
r = (vs - vt).dot(nt);
A_r.block<3, 1>(0, 0) = vs.cross(nt);
A_r.block<3, 1>(3, 0) = nt;
ATA += A_r * A_r.transpose();
ATb += A_r * r;
}
tend = time(0);
std::cout << "Option1 took " << difftime(tend, tstart) << " second(s)." << std::endl;
// Option #2
// Seperately build J and r
tstart = time(0);
Eigen::MatrixXd A(NCORR,6);
Eigen::VectorXd b(NCORR);
A.setZero();
b.setZero();
for (size_t i = 0; i < NCORR; i++) {
const Eigen::Vector3d &vs = data[i];
const Eigen::Vector3d &vt = data[i];
const Eigen::Vector3d &nt = data[i];
r = (vs - vt).dot(nt);
A_r.block<3, 1>(0, 0) = vs.cross(nt);
A_r.block<3, 1>(3, 0) = nt;
A.block<1, 6>(i, 0) = A_r;
b(i) = r;
}
Eigen::MatrixXd AT = A.transpose();
ATA = AT * A;
ATb = AT * b;
tend = time(0);
std::cout << "Option2 took " << difftime(tend, tstart) << " second(s)." << std::endl;
The outputs are
Option1 took 6 second(s).
Option2 took 12 second(s).
It is obvious Option 1 is faster than Option 2. It is presumably due to better cash hit.
Option 1 consumes much smaller amount of memory as well.
However, the question arises when the for
loop is paralleled using OpenMP
.
// Option #3
// Directly Building JtJ and Jtr
tstart = time(0);
ATA.setZero();
ATb.setZero();
#ifdef _OPENMP
#pragma omp parallel for schedule(static)
#endif
for (int i = 0; i < NCORR; i++) {
const Eigen::Vector3d &vs = data[i];
const Eigen::Vector3d &vt = data[i];
const Eigen::Vector3d &nt = data[i];
double r = (vs - vt).dot(nt);
Eigen::Vector6d A_r;
A_r.block<3, 1>(0, 0) = vs.cross(nt);
A_r.block<3, 1>(3, 0) = nt;
#ifdef _OPENMP
#pragma omp critical
#endif
{
ATA += A_r * A_r.transpose();
ATb += A_r * r;
}
}
tend = time(0);
std::cout << "Option3 took " << difftime(tend, tstart) << " second(s)." << std::endl;
// Option #4
// Seperately build J and r
tstart = time(0);
A.setZero();
b.setZero();
#ifdef _OPENMP
#pragma omp parallel for schedule(static)
#endif
for (int i = 0; i < NCORR; i++) {
const Eigen::Vector3d &vs = data[i];
const Eigen::Vector3d &vt = data[i];
const Eigen::Vector3d &nt = data[i];
r = (vs - vt).dot(nt);
Eigen::Vector6d A_r;
A_r.block<3, 1>(0, 0) = vs.cross(nt);
A_r.block<3, 1>(3, 0) = nt;
A.block<1, 6>(i, 0) = A_r;
b(i) = r;
}
AT = A.transpose();
ATA = AT * A;
ATb = AT * b;
tend = time(0);
std::cout << "Option4 took " << difftime(tend, tstart) << " second(s)." << std::endl;
The output is
Option3 took 43 second(s).
Option4 took 6 second(s).
I had to put omp critical
for Option 3 because ATA
and ATb
should not be accumulated by multiple threads at the same time. Without omp critical
, it makes unreliable sum. omp critical
actually mess up because only single thread gets writing authorization and other threads need to wait until the writing business is done.
This experiment indicates that how we write parallel code using Eigen::MatrixXd
, and indicates why parallel implementation of RGBD odometry is not that fast (related to #51) compared to original code. The Jacobian module in RGBD Odometry is similar to Option 4, and the original implementation is similar to Option 1. As we observed, Option 1 and Option 4 takes 6 seconds.
Do you have any idea how we can write more effective code? I am individually investigating now. The goal is to design 'option 5' that takes just 1~2 seconds.
Hi guys,
I just wanted to let you know that I just started to seriously work with Open3D and it is absolutely awesome. I especially like the python bindings!
That said I wanted to point out that CamelCase for function names is very unusual in the python world. The standard style guide https://www.python.org/dev/peps/pep-0008/#function-names suggest lower case with underscores. AFAIK most python packages follow this convention, and I could imagine users asking for this pretty quickly after the initial release.
Follow up work of #46
These will be the next pull request.
Additionally, the next step is FastGlobalRegistration. It is another PR.
As discussed in #42 , in the future, we should add conversion function to covert different size and different view point RGB-D image pairs. Quote:
There are some minor formatting issues. Others are good.
I have an additional question: what if the input color image is of different size of depth image? There are some use cases:
We don't need to handle them immediately. But I think we should leave a note at the class definition of RGBDImage, saying that it is for a pair of registered color and depth images, viewed from the same view, of the same resolution. If you have other format, convert it first.
In the future, we can provide similar helper functions for other kinds of input (as long as the input resolution and the calibrated extrinsic matrix is given). But it is future work.
The current pybind11 is modified from 2.0 dev.
The main modifications are:
std::vector
binding styleThe first modification is now merged into pybind11 master branch.
The pybind11 in Open3D should be upgraded to mainstream stable pybind11 version.
I am trying to implant our ICCV code to Open3D.
The files I am editing are core/registration/TransformationEstimation.h/cpp and Registration.cpp. The question is the following virtual function:
Eigen::Matrix4d ComputeTransformation(const PointCloud &source,
const PointCloud &target,
const CorrespondenceSet &corres) const override;
for colored points registration, we need additional argument e.g. point-wise color gradient for computing Jacobian. Any good idea for this?
In addition, we know the multi-scale registration really helps registration fidelity. Our ICCV implementation code is based on this idea, but I would like to make a new wrapper class so all the ICP methods in Open3D can get benefits. How do you think?
Call the following command from OSX console.
python test_py3d.py visualization
The glfw window does not receive any keyboard input.
#41 covers basic data structure and functions for RGBD odometry application.
There are few issues additionally need to be handled.
as title.
This can be useful in scenarios where users want to render a (long) video.
It can also be useful when Open3D is deployed in a remote server and window creation is a problem.
Some information regarding GLFW: glfw/glfw#648
Need to add Visual Studio 2017 into the tool chain.
Later on, we should setup travis-ci or similar testing systems to test the build of Visual Studio.
https://learnopengl.com/#!In-Practice/Text-Rendering
https://learnopengl.com/code_viewer.php?code=in-practice/text_rendering
This should go into a Misc category as one of ShaderWrapper's sub-classes.
Main roadblock:
FreeType is a complicated library to compile with.
This is also related to #1
Things like:
SelectionPolygonVolume::CropGeometry
SelectDownSample
, VoxelDownSample
, UniformDownSample
, ...ComputePointCloudToPointCloudDistance
, ...Move the output to the return part of functions. For all Geometry
type, return std::share_ptr
because Visualizer
needs to share the ownership. But for std::vector
, use the move and trivial copy mechanics to directly return.
There is a warning when I compiling Open3D with OSX-El capitan.
/Users/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FileJSON.cpp:42:8: warning: 'Reader' is deprecated: Use CharReader and CharReaderBuilder instead [-Wdeprecated-declarations]
Json::Reader reader;
^
/usr/local/Cellar/jsoncpp/1.8.3/include/json/reader.h:35:83: note: 'Reader' has been explicitly marked deprecated here
class JSONCPP_DEPRECATED("Use CharReader and CharReaderBuilder instead") JSON_API Reader {
^
/Users/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FileJSON.cpp:59:8: warning: 'StyledStreamWriter' is deprecated: Use StreamWriterBuilder instead [-Wdeprecated-declarations]
Json::StyledStreamWriter writer;
^
/usr/local/Cellar/jsoncpp/1.8.3/include/json/writer.h:273:70: note: 'StyledStreamWriter' has been explicitly marked deprecated here
class JSONCPP_DEPRECATED("Use StreamWriterBuilder instead") JSON_API StyledStreamWriter {
^
Can we replace Reader
as suggested, or upgrade jsoncpp?
#64 works reasonably well, but it need to be run faster.
I am working on this.
We have flexible Jacobian evaluation for RGBDOdometry in Core/Odometry/RGBDOdometryJacobian.cpp
. However, using std::vector<Eigen::Vector6d>
has memory alignment issue. According to Eigen Issue, it says:
Otherwise, without knowing the specialization the compiler will compile that particular instance with the default std::allocator and you program is most likely to crash.
Actually this is happening to my Ubuntu 16.04. The odometry module sometimes crashes with segfault due to this issue.
Option 1
I tried two remedies described in the Eigen Issue, but could not resolve the problem. This also depends on how to interpret assignment operator by C++11 compilers (see the answer in this link.) To my understanding at this moment, this option seems to be not valid.
Option 2
Another way is not using std::vector<Eigen::Vector6d>
, but this will break the structure of virtual class ComputeJacobianAndResidual
as its inherited classes: RGBDOdometryJacobianFromColorTerm
and RGBDOdometryJacobianFromHybridTerm
as they take different size of Jacobian matrix (6x1 matrix and 6x2 matrix, respectively).
Do you have any suggestion on this?
This is a must have for Open3D.
I looked around, the current best solution seems to be pybind11.
https://github.com/pybind/pybind11
It is neat, header only, cross platform, and supports any non-ancient C++ compiler.
Main roadblock: I need to find time to do it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.