lmb-freiburg / demon Goto Github PK
View Code? Open in Web Editor NEWDeMoN: Depth and Motion Network
Home Page: https://lmb.informatik.uni-freiburg.de/people/ummenhof/depthmotionnet/
License: GNU General Public License v3.0
DeMoN: Depth and Motion Network
Home Page: https://lmb.informatik.uni-freiburg.de/people/ummenhof/depthmotionnet/
License: GNU General Public License v3.0
when I run python3 example.py,I get "module 'lmbspecialops' has no attribute 'leaky_relu'"
I'm currently testing the tool using the example.py with sculpture1 and 2. I'm running it with CPU and getting the following error statement:
2017-09-05 11:01:05.625707: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-05 11:01:05.625729: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-05 11:01:05.625739: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
In file included from /usr/include/numpy/ndarraytypes.h:1761:0,
from /usr/include/numpy/ndarrayobject.h:17,
from /usr/include/numpy/arrayobject.h:4,
from /home/zurich/.pyxbld/temp.linux-x86_64-3.5/pyrex/depthmotionnet/vis_cython.c:485:
/usr/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
Traceback (most recent call last):
File "example.py", line 112, in <module>
translation=translation)
File "../python/depthmotionnet/vis.py", line 280, in visualize_prediction
pointcloud = compute_point_cloud_from_depthmap(depth, K, R1, t1, n, img)
File "../python/depthmotionnet/vis.py", line 47, in compute_point_cloud_from_depthmap
return _compute_point_cloud_from_depthmap(depth, K, R, t, normals, colors)
File "vis_cython.pyx", line 171, in depthmotionnet.vis_cython.compute_point_cloud_from_depthmap
ValueError: shape mismatch: colors (3, 256, 6), depth (192, 256)
I'm using an Anaconda environment with:
The Depth map I get is correct, but when it gets to the point cloud the error statement appears.
Thank you in andvanced.
Best regards,
Zuria.
In the examples there're files: sculpture_Rt1.txt and sculpture_Rt2.txt with camera positions.
I cannot find a place where you're actually calculating those positions - also they're not updating after running example.py.
Could you explain how to use function for calculating them properly ?
Also - I am planning to generate a series of depthmaps from the same scene - and would like to have a camera positions for each of them relatively to each other in one coordinate system - any tips, what's best approach to do that ?
Thanks for sharing the work, great and impressive piece of code.
Hello.
Thanks for the code.
I have a question about the data reader.
Is the image distortion is pre-distorted when build the hdf5 file for rgbd-slam dataset ?
Thanks.
I'm currently using rgbd_dataset_freiburg1_xyz as input of the example code. However, the output of the depth map prediction is so dark that i cant even see the architecture of the scene comparing to the ground truth provided by the dataset. Is the output depth map of the example code inverted? or it is the actual depth? also if i wish to compare the depth map to the ground truth, i will need to scale up the output of the example, how can i obtain the scale between the output and the ground truth?
Hi,
When I run python3 create_dataset_and_use_readerop.py , an errror message showed up:
Traceback (most recent call last):
File "../python/depthmotionnet/dataset_tools/lz4.py", line 24, in
liblz4 = CDLL(_lib_path)
File "/usr/lib/python3.5/ctypes/init.py", line 347, in init
self._handle = _dlopen(self._name, mode)
OSError: /media/deep/文档/ubuntubf/test/demon/build/lz4/src/lz4-build/liblz4.so: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "../python/depthmotionnet/dataset_tools/lz4.py", line 28, in
liblz4 = CDLL('liblz4.so')
File "/usr/lib/python3.5/ctypes/init.py", line 347, in init
self._handle = _dlopen(self._name, mode)
OSError: liblz4.so: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "create_dataset_and_use_readerop.py", line 12, in
from depthmotionnet.dataset_tools import *
File "../python/depthmotionnet/dataset_tools/init.py", line 19, in
from .view_io import *
File "../python/depthmotionnet/dataset_tools/view_io.py", line 21, in
from .lz4 import lz4_uncompress, lz4_compress_HC
File "../python/depthmotionnet/dataset_tools/lz4.py", line 30, in
raise RuntimeError('Cannot load liblz4.so')
RuntimeError: Cannot load liblz4.so
Any ideas?
Thanks,
Zhang-jn
What is the physical unit of the final depth map? Is there a easy way to obtain a depth map e.g. in meters?
I guess one has to first invert the inverse depth back to the physical depth. But then it still needs to be converted into SI-units. I tried to eyeball the conversion factor, but could not find a consistent number.
I could imagine that the scale factor (the s from in the paper) cannot be estimated very accurately. Just imagine having two image pairs: one of a usual lego figure, and one of a human sized one. There would be no way to tell them apart right?
I generated solution file with cmake and tried building the code with visual studio (2010, 2012, 2013), but I am getting some errors:
Error 1 error MSB6006: "cmd.exe" exited with code 9009. C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets 170 5 webp
Error 2 error D8021: invalid numeric argument '/Wabi-tag' E:\Vijay Bhardwaj\github repos\demon-master-new\lmbspecialops\build\multivih5datareaderop\cl multivih5datareaderop
When I looked at the output window, it shows that
'CFLAGS' is not recognized as an internal or external command,
operable program or batch file.
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(170,5): error MSB6006: "cmd.exe" exited with code 9009.
Another line says:
cl : Command line error D8021: invalid numeric argument '/Wabi-tag'
Can you help me out with this??
I am wondering is there are training script for this as well. Thank you.
Panorma has more context, but what is the best practice to get the depth map for 2 paired panorama? should we change it to cubemap and estimate each face and stitch them together.. NOt sure, if the individual face is consistent with each other's result. Or should we consider directly use the distorted panorama for training?
Hello!
From the sculpture example, I was able to deduce that the network is predicting egomotion in a coordinate system where +x points left, +y points up, +z points backwards. Is this correct? If so, I'm curious why this system was chosen. I'm not sure if there's a "standard", I just noticed that vkitti uses +x right, +y down, +z forward.
Best,
Sam
Hello,
I am trying to finetune your iterative network with my own dataset.
I understand the ground truth must be:
However, at the start of training, I'm comparing the ranges of the ground truth to predictions from the iterative net:
[name][min][max]
[gt depth][0.0015259022][0.25714538]
[pred depth][-4.7255404e+09][4.8389247e+10]
[gt flow][-0.33519724][0.31490907]
[pred flow][-1.160358][0.12831117]
[gt normals][-1][0.99998885]
[pred normals][-3502147.5][2898221.5]
The ranges look reasonable for flow, however the predictions for depth and normals are much larger than the gt. I am generating the gt normals using the depth_to_normals function from lmbspecialops with my gt inverse depth.
Am I missing something with my preprocessing? For reference, I'm using the virtual kitti dataset.
Thank you for your time,
Sam
I think the scaling factor can be trained like rotation and translation, because the true scaling factor can be obtained by normalizing translation when training, is it right? But in the DeMoN paper, it seems the scale is only trained together with the inverse depth.
Hi Just wonder what kind of data the release model is trained on ?
Is it also possible to have other models about your prediction of different scenes ? Or you use the same model for both indoor and outdoor images ?
Thanks
hello,
I wonder that if you use the Matlab toolbox provided by nyuv2 website to impaint holes in raw depth map, or you just use raw depth staightforward?
thank you~
Hi,
When I run python3 example.py
, an errror message showed up:
Traceback (most recent call last):
File "example.py", line 12, in
from depthmotionnet.networks_original import *
File "../python/depthmotionnet/networks_original.py", line 18, in
from .blocks_original import *
File "../python/depthmotionnet/blocks_original.py", line 19, in
from .helpers import *
File "../python/depthmotionnet/helpers.py", line 19, in
import lmbspecialops as sops
File "/home/bingyao/demon/lmbspecialops/python/lmbspecialops.py", line 30, in
lmbspecialopslib = tf.load_op_library(_lib_path)
File "/home/bingyao/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File "/home/bingyao/.local/share/virtualenvs/demon_venv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /home/bingyao/demon/lmbspecialops/build/lib/lmbspecialops.so: undefined symbol: _ZTIN10tensorflow8OpKernelE
Any ideas?
Thanks,
Bingyao
I'm using Demon to compare it with other Depthmaps and I have some trouble with the units of the output of Demon. Could you tell me in which units is Demon operating?
Hello @benjaminum,
I want to train the network from scratch, is it possible to get the train code?
Hello!
I want to finetune your network on my dataset. I have a couple questions about how you initially trained the network.
First you trained the four encoder-decoders in the bootstrap and iterative net sequentially for 250k iterations. This is straightforward.
Second you trained the iterative net for 1600k iterations. You appended the predictions of the iterative net to the inputs to the iterative net to emulate having 4x iterative nets while only having one net in memory.
Finally, do you see any need to finetune the refinement net or is it sufficient to finetune the iterative net with simulated iterations?
Thank you!
Sam Pepose
While building lmbspecialops:
[ 33%] No install step for 'lz4'
[ 35%] Completed 'lz4'
[ 35%] Built target lz4
[ 37%] Building NVCC (Device) object lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_median3x3downsample_cuda.cu.o
CMake Error at lmbspecialops_generated_median3x3downsample_cuda.cu.o.cmake:207 (message):
Error generating
/home/sharkubuntu/demon/lmbspecialops/build/lib/CMakeFiles/lmbspecialops.dir//./lmbspecialops_generated_median3x3downsample_cuda.cu.o
lib/CMakeFiles/lmbspecialops.dir/build.make:112: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_median3x3downsample_cuda.cu.o' failed
make[2]: *** [lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_median3x3downsample_cuda.cu.o] Error 1
CMakeFiles/Makefile2:164: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/all' failed
make[1]: *** [lib/CMakeFiles/lmbspecialops.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2
python3 -c 'import tensorflow as tf; print(tf.version)':
1.6.0
nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
lsb_release -a:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
cmake --version:
cmake version 3.5.1
CMake suite maintained and supported by Kitware (kitware.com/cmake).
Hello @benjaminum ,
Thank you and your partners so much about the greate Computer Vision work.
Today, I have tried to build your branch master
with Ubuntu 18.04, after some tweaks with CMake (I disabled CUDA as I don't have NVidia GPU available and link the Python path to Python 3.6.5, TENSORFLOW_LIB_DIR to tensorflow_framework.so as I use Tensorflow 1.8 binary with naive pip).
But the make
stage show me error on
#include "tensorflow/core/framework/op.h"
Is .so
file I linked is incorrect and if so, which file I have to link with?
And if not, could you please tell me the solution?
Thanks in advance.
In the paper, it states one of output of the fully connected layers is a depth scale factor s. The scale factor s relates the scale of the depth values to the camera motion. Ref: Figure 3 in the paper.
Looking at the example code and then the source code, I am unable to figure out how I can get the scale factor value. What am I missing? Any help appreciated.
The image pairs used in training don't look like the stereo pairs. Are they consecutive frames ?
How are these 'unconstrained image pairs' considered for training?
Also for prediction, image pairs are required in the same manner?
Hi,
Thank you for your last help. Maybe this problem needs your help again this time.
When I run python training.py, an errror message showed up:
Traceback (most recent call last):
...
Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(16, 4608), dtype=float32, device=/device:GPU:0)
Tensor("tower_1/netFlow1/dense5/LeakyRelu:0", shape=(16, 4608), dtype=float32, device=/device:GPU:1)
nothing to restore. no checkpoint found.
train evolution "0_flow1" ...
start iteration 0
2017-12-19 22:36:48.508170: I tensorflow/stream_executor/dso_loader.cc:139] successfully opened CUDA library libcupti.so.8.0 locally
Segmentation fault(core dumped)
Any ideas?
Thanks,
JUNZHANG
tensorflow-gpu 1.0.0
python 3.5
...
All the libs satisfied with the requirements specified in readme.
But got errors in the make step.
'demon/lmbspecialops/src/decode_lz4_op.cc:43:29: error: ‘class tensorflow::shape_inference::InferenceContext’ has no member named ‘MakeShapeFromPartialTensorShape’
TF_RETURN_IF_ERROR(c->MakeShapeFromPartialTensorShape(expected_shape, &out));'
After around 10 days of training, the script generates the snapshot-250000.data-00000-of-00001 of the refine stage. However, plugging these newly computed weights to the provided example.py script I am getting errors starting with Not found: Key netDM1/motion_conv1/kernel not found in checkpoint...
:
The only suspicious thing I saw during training is lots of warnings on empty queues or something like that.. but these seem to go away after a couple of stages of training.
MultiGPU training with netFlow1
works fine to me, while for the next evolution training some tensors are shown 'None values'.
I'm running on a machine installed with 4 GPUs, while only feed the program two of them by CUDA_VISIBLE_DEVICES=2,3
. I noticed that get_gpu_count()
did return 4 even two gpus are fed into. I tried set _num_gpus = 2
manually but this didn't solve the problem.
BTW: Did you not divide loss by batch size for some reason on purpose? or this is just another normal way to deal with loss and batch size..
Thanks!
➜ v2 git:(devel) ✗ CUDA_VISIBLE_DEVICES=2,3 ipython training-testDM-multiGPU.py
Using /home/jiamingsun/Repositories/demon/build/multivih5datareaderop/multivih5datareaderop.so
Using /home/jiamingsun/Repositories/demon/lmbspecialops/build/lib/lmbspecialops.so
2017-10-11 17:01:19.298226: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298265: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298273: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298281: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.298288: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-10-11 17:01:19.875586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:83:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-10-11 17:01:19.875754: W tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x3e92220
2017-10-11 17:01:20.206920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 1 with properties:
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:84:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-10-11 17:01:20.208079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 1
2017-10-11 17:01:20.208094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y Y
2017-10-11 17:01:20.208100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1: Y Y
2017-10-11 17:01:20.208119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:83:00.0)
2017-10-11 17:01:20.208128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:84:00.0)
Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:0)
Tensor("tower_0/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:0)
[<tf.Tensor 'tower_0/gradients/tower_0/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_0/gradients/tower_0/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
Tensor("tower_1/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:1)
Tensor("tower_1/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:1)
[<tf.Tensor 'tower_1/gradients/tower_1/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_1/gradients/tower_1/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
Tensor("tower_2/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:2)
Tensor("tower_2/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:2)
[<tf.Tensor 'tower_2/gradients/tower_2/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_2/gradients/tower_2/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
Tensor("tower_3/netFlow1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:3)
Tensor("tower_3/netDM1/dense5/LeakyRelu:0", shape=(8, 4608), dtype=float32, device=/device:GPU:3)
[<tf.Tensor 'tower_3/gradients/tower_3/netDM1/predict_depthnormal2/split_grad/concat:0' shape=(8, 4, 48, 64) dtype=float32>, None, None]
[<tf.Tensor 'tower_3/gradients/tower_3/netDM1/split_grad/concat:0' shape=(8, 7) dtype=float32>, None, None]
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
490 as_ref=input_arg.is_ref,
--> 491 preferred_dtype=default_dtype)
492 except TypeError as err:
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype)
703 if ret is None:
--> 704 ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
705
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
112 _ = as_ref
--> 113 return constant(v, dtype=dtype, name=name)
114
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
101 tensor_value.tensor.CopyFrom(
--> 102 tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
103 dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
359 if values is None:
--> 360 raise ValueError("None values not supported.")
361 # if dtype is provided, forces numpy array to be the type
ValueError: None values not supported.
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
504 observed = ops.internal_convert_to_tensor(
--> 505 values, as_ref=input_arg.is_ref).dtype.name
506 except ValueError as err:
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype)
703 if ret is None:
--> 704 ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
705
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
112 _ = as_ref
--> 113 return constant(v, dtype=dtype, name=name)
114
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
101 tensor_value.tensor.CopyFrom(
--> 102 tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
103 dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
359 if values is None:
--> 360 raise ValueError("None values not supported.")
361 # if dtype is provided, forces numpy array to be the type
ValueError: None values not supported.
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
/home/jiamingsun/Repositories/demon/training/v2/training-testDM-multiGPU.py in <module>()
619
620 if __name__ == '__main__':
--> 621 tf.app.run()
622
623
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py in run(main, argv)
46 # Call the main function, passing through any arguments
47 # to the final program.
---> 48 _sys.exit(main(_sys.argv[:1] + flags_passthrough))
49
50
/home/jiamingsun/Repositories/demon/training/v2/training-testDM-multiGPU.py in main(argv)
558
559 # combine gradients from all towers
--> 560 avg_grads = average_gradients(tower_grads)
561
562 optimize_op = optimizer.apply_gradients(grads_and_vars=avg_grads, global_step=trainer.global_step())
/home/jiamingsun/Repositories/demon/python/tfutils/helpers.py in average_gradients(tower_grads)
276 for g, _ in grad_and_vars:
277 # Add 0 dimension to the gradients to represent the tower.
--> 278 expanded_g = tf.expand_dims(g, 0)
279
280 # Append on a 'tower' dimension which we will average over below.
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py in expand_dims(input, axis, name, dim)
168 raise ValueError("can't specify both 'dim' and 'axis'")
169 axis = dim
--> 170 return gen_array_ops._expand_dims(input, axis, name)
171 # pylint: enable=redefined-builtin,protected-access
172
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py in _expand_dims(input, dim, name)
898 dimension of size 1 added.
899 """
--> 900 result = _op_def_lib.apply_op("ExpandDims", input=input, dim=dim, name=name)
901 return result
902
/home/jiamingsun/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
507 raise ValueError(
508 "Tried to convert '%s' to a tensor and failed. Error: %s" %
--> 509 (input_name, err))
510 prefix = ("Input '%s' of '%s' Op has type %s that does not match" %
511 (input_name, op_type_name, observed))
ValueError: Tried to convert 'input' to a tensor and failed. Error: None values not supported.
Just to note VTK for Python 3 is available with Anaconda
$ conda search vtk
Fetching package metadata ...........
pyvtk 0.5.18 py27_0 conda-forge
0.5.18 py35_0 conda-forge
0.5.18 py36_0 conda-forge
vtk 7.0.0 py27_0 conda-forge
7.0.0 py35_0 conda-forge
7.0.0 py27_1 conda-forge
7.0.0 py35_1 conda-forge
7.0.0 py27_2 conda-forge
7.0.0 py35_2 conda-forge
7.1.0 py27_0 conda-forge
7.1.0 py35_0 conda-forge
7.1.0 py27_1 conda-forge
7.1.0 py35_1 conda-forge
7.1.0 py27_2 conda-forge
7.1.0 py35_2 conda-forge
7.1.0 py27_3 conda-forge
7.1.0 py35_3 conda-forge
7.1.0 py36_3 conda-forge
7.1.0 py27_4 conda-forge
7.1.0 py35_4 conda-forge
7.1.0 py36_4 conda-forge
5.10.1 py27_0 defaults
5.10.1 py27_1 defaults
6.3.0 py27_0 defaults
6.3.0 py27_1 defaults
Thank you for making the code available! Is there some example code for training the network available?
I want to learn the training network of this model and make some improvements on it, but I do not know how to train. Can you open source the training network? Thanks~
Did you do any data augmentation (DA) for training DeMoN just as FlowNet2? It seems a little difficult to performe DA because of different kinds of inputs, such as flow, warped image and depth. If you did DA, could you give me some advice about how to perform DA for DeMoN?
Hi @benjaminum, have you ever successfully trained the network with your latest v2 training code? I've been experiment with it for a few days, finding it's extremely hard to converge.
This is my training loss status:
(please ignore TotalLoss_1)
This is trained within about 500k iterations with default start learning rate and batch size on the first evolution. I also tried other lr and optimizer configurations, the result is similar.
blocks.py
, could that be a reason? Why would you make this change and what is v2 means exactly..Thanks for this amazing work!
Hi, when I run training.py, I got such error message, any idea please.
by the way, the tensorflow version is 1.4.0, not compiled.
<module 'depthmotionnet.datareader' from '/home/kylin/demon/python/depthmotionnet/datareader/init.py'>
Tensor("tower_0/netFlow1/dense5/LeakyRelu:0", shape=(32, 4608), dtype=float32, device=/device:GPU:0)
nothing to restore. no checkpoint found.
train evolution "0_flow1" ...
start iteration 0
2018-05-17 19:37:59.029835: E tensorflow/core/common_runtime/executor.cc:643] Executor failed to create kernel. Invalid argument: Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]
stop iteration 0
Traceback (most recent call last):
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "training.py", line 604, in
tf.app.run()
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "training.py", line 598, in main
trainer.coordinator().raise_requested_exception()
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 407, in raise_requested_exception
six.reraise(*self._exc_info_to_raise)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/six.py", line 693, in reraise
raise value
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 296, in stop_on_exception
yield
File "/home/kylin/Applications/tfutils/python/tfutils/simpletrainer.py", line 280, in mainloop
values = self._session.run(ops, options=run_options, run_metadata=run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]
Caused by op 'tower_0/netFlow1/refine4/upconv/conv2d_transpose', defined at:
File "training.py", line 604, in
tf.app.run()
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "training.py", line 193, in main
kernel_regularizer=_kernel_regularizer,
File "/home/kylin/demon/python/depthmotionnet/v2/blocks.py", line 231, in flow_block
**conv_params,
File "/home/kylin/demon/python/depthmotionnet/v2/blocks.py", line 108, in _refine
**kwargs,
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1423, in conv2d_transpose
return layer.apply(inputs)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 671, in apply
return self.call(inputs, *args, **kwargs)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 575, in call
outputs = self.call(inputs, *args, **kwargs)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/layers/convolutional.py", line 1289, in call
data_format=utils.convert_data_format(self.data_format, ndim=4))
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1211, in conv2d_transpose
name=name)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 779, in conv2d_backprop_input
use_cudnn_on_gpu=use_cudnn_on_gpu, data_format=data_format, name=name)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/kylin/.local/share/virtualenvs/demon_venv/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInputOp only supports NHWC.
[[Node: tower_0/netFlow1/refine4/upconv/conv2d_transpose = Conv2DBackpropInput[T=DT_FLOAT, data_format="NCHW", padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/gradients/tower_0/netFlow1/refine4/concat_grad/Shape, netFlow1/refine4/upconv/kernel/read, tower_0/netFlow1/concat)]]
I try to follow the readme but cannot get it right. I am using tensorflow 1.4.0 with python 3.5.2, on ubuntu 16.04, here is the error message for python3:
demon_venv tianwei@tianwei:~/Documents/Projects/demon/examples$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.import lmbspecialops
Traceback (most recent call last):
File "", line 1, in
File "/home/tianwei/Documents/Projects/demon/lmbspecialops/python/lmbspecialops.py", line 30, in
lmbspecialopslib = tf.load_op_library(_lib_path)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: /home/tianwei/Documents/Projects/demon/lmbspecialops/build/lib/lmbspecialops.so: undefined symbol: _ZN10tensorflow11GetNodeAttrERKNS_9AttrSliceENS_11StringPieceEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Hi, I have test your vtk code and create point clouds of one single image (fr3_teddy).
But I cannot get quality result if input multi-images (use your rgbd_test.h5 file).
Have you reconstruction 3D point clouds use multi-images?
Can you send me your reconstruction code if you have done this work?
My email address [email protected]. Thks a lot
Here are my test results:
Are there some errors in my test?
I am trying to run demon against a set of images I extracted from a video passing them into bootstrap_net.eval. The depth images I get back are not smooth from one image to the next but have a lot of inconsistency between pairs. The project web site has a video of a road with a bicyclist that looks like it was webcam or video. I normalize the depth images but this did not help. How was the project video made?
File "create_dataset_and_use_readerop.py", line 79, in
reader_tensors = multi_vi_h5_data_reader(len(data_tensors_keys), json.dumps(reader_params))
TypeError: 'NoneType' object is not callable
Dear,
I tried to evaluate DeMoN using l1-inv and l1-rel as stated in the paper. However, I found the value does not consist with the paper:
1, the output depth is scaled or not? it equals to the depth when the norm translation equals to 1.0?
2, Even using S_{log} as stated in the supplementary material, l1-inv is still bigger than paper reported.
One difference is that I evaluate all the pixels, instead of two-frame visible pixels. Is this a big difference?
Thanks for this great work. Just one important question. I would like to try this with images using an intrinsic matrix, let's call this K. I see that the network was trained using a fixed intrinsic matrix, let's call it Khat.
The way to do this would seem to be to warp my images by a corrective homography H = Khat*inv(K). Then I can pass the warped images to the network. This seems fine, but the main problem with this is when K has a longer focal length than Khat, my warped image will need to be padded at the borders. Standard padding techniques (zero padding) gives very bad results.
Can you see any simple way that this can be resolved? It's related to you mentioning cropping as a way to use a different intrinsic matrix, however if the focal length is longer we can't deal with it by cropping.
Much appreciated for your great paper and code, Toby
Hello, thanks for your sharing with demon.
I have already run your evaluation.py successfully, and result shows your method very effective.
Now I want to create a new dataset(such as rgbd slam download from tum) and train this new dataset use your project, what should I do?
Could you give whole source code of training ?
Could you give list of rgbd-slam dataset which you used for training and dataset select code ?
Thanks again !
Hi, in the paper, you wrote "We resolve the scale ambiguity by normalizing translations and depth values such that ||t||=1." Now, I want to obtain actual translation (not normalized). Is it possible?
To obtain the real depth map , I invert the pixel value of the output depth map, multiply it with the scaling factor s. However, the result is not very good. I wonder if I have made some mistakes here?
HI Benjamin,
Bug in your depth2normal code ? I recently use your depth2normal code and find the tensorflow implementation.
const T cx = 1/(intrinsics[4*z+2]*x_size);
const T cy = 1/(intrinsics[4*z+3]*y_size);
you have cx, cy as inverse of the input. I think it might be a bug since when we compute cx, cy, we do not need to inverse it.
Hi, Is it possible to train the DEMON using CPU??
If yes, what changes I need to do in the code??
Hi, thanks for such an interesting work and making DeMoN open source.
I have a question regarding the output format of the relative poses predicted in the example.py file. It is clear that translation is a normalised unit vector with 3 elements. But for the rotation prediction, it is also a 3-element vector. What are these values? How does the vector represent the rotation, as mentioned in the paper " The rotation r = θv is an angle axis representation with angle θ and axis v". I just want to convert it to a rotation matrix. Would you please share more details? Thanks a lot!
I've been trying to run this with a webcam and have been getting very odd segment faults in tensorflow. I was wondering if you have tried anything like this.
the basic code loop is:
img1 = Image.open(os.path.join(examples_dir,'sculpture2.png'))
img2 = Image.open(os.path.join(examples_dir,'sculpture1.png'))
video = cv2.VideoCapture('/home/rjn/data/videos/city.mp4')
while(True):
ok, buf = video.read()
buf = cv2.cvtColor(buf,cv2.COLOR_BGR2RGB)
img2 = Image.fromarray(buf)
*
*
*
img1 = img2.copy()
Hello,
Is there a simple way to apply DeMoN to batches of images? Recent GPUs have plenty of memory and inference could be sped up by spending less time moving data around. There is a hardcoded batch size of 1, for example here ff. So just pushing batches of data into DeMoN does not work, sadly.
I am wondering how complex the required changes are. Are there any implementation details that forbid batched inference? Maybe you can give some hints about where non trivial changes are neccessary
Thanks
hello,
when i make,i have some errors as follows.
thank you for your answer.
[ 37%] Building NVCC (Device) object lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_warp2d_cuda.cu.o
/home/oem/demon/lmbspecialops/src/warp2d_cuda.cu:20:49: fatal error: tensorflow/core/framework/op_kernel.h: No such file or directory
compilation terminated.
CMake Error at lmbspecialops_generated_warp2d_cuda.cu.o.Release.cmake:222 (message):
Error generating
/home/oem/demon/lmbspecialops/build/lib/CMakeFiles/lmbspecialops.dir//./lmbspecialops_generated_warp2d_cuda.cu.o
lib/CMakeFiles/lmbspecialops.dir/build.make:112: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_warp2d_cuda.cu.o' failed
make[2]: *** [lib/CMakeFiles/lmbspecialops.dir/lmbspecialops_generated_warp2d_cuda.cu.o] Error 1
CMakeFiles/Makefile2:127: recipe for target 'lib/CMakeFiles/lmbspecialops.dir/all' failed
make[1]: *** [lib/CMakeFiles/lmbspecialops.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2
CMake Error at /usr/share/cmake-3.5/Modules/ExternalProject.cmake:1888 (message):
Could not extract tarball filename from url:
https://sourceforge.net/projects/half/files/latest/download?source=files
Call Stack (most recent call first):
/usr/share/cmake-3.5/Modules/ExternalProject.cmake:2459 (_ep_add_download_command)
multivih5datareaderop/CMakeLists.txt:55 (ExternalProject_Add)
Any suggestions for how to solve this?
Could you give an example on how to use the intrinsics to crop and image as stated in example. Not sure how to proceed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.