ERROR : Run demo.py --gpu 0 about person_search

[Cysu]

@jhj7905 Could you please attach the full error report?

[jhj7905]

@Cysu i attached the error report below

thank you

I0830 11:22:04.826299 8202 net.cpp:1272] blob 169 name det_score diff idx -1
I0830 11:22:04.826318 8202 net.cpp:1262] blob 170 name pid_prob data idx -1
I0830 11:22:04.826335 8202 net.cpp:1272] blob 170 name pid_prob diff idx -1
I0830 11:22:04.826354 8202 net.cpp:1262] blob 171 name bbox_pred data idx -1
I0830 11:22:04.826371 8202 net.cpp:1272] blob 171 name bbox_pred diff idx -1
I0830 11:22:04.826390 8202 net.cpp:1262] blob 172 name feat_lowdim data idx 0
I0830 11:22:04.826409 8202 net.cpp:1272] blob 172 name feat_lowdim diff idx 0
I0830 11:22:04.826427 8202 net.cpp:1262] blob 173 name feat data idx -1
I0830 11:22:04.826445 8202 net.cpp:1272] blob 173 name feat diff idx -1
I0830 11:22:04.826462 8202 net.cpp:1278] storage memory slot 0 size 3211264
I0830 11:22:04.826479 8202 net.cpp:1278] storage memory slot 1 size 3211264
I0830 11:22:04.826496 8202 net.cpp:1278] storage memory slot 2 size 3211264
I0830 11:22:04.826514 8202 net.cpp:1278] storage memory slot 3 size 401408
I0830 11:22:04.826530 8202 net.cpp:1283] raw memory 310126112 opt memory 11241632
I0830 11:22:34.060151 8202 cudnn_conv_layer.cpp:186] Optimized cudnn conv
demo/image2.jpg ...
F0830 11:22:34.267608 8202 cudnn_conv_layer.cu:33] Check failed: status == CUDNN_STATUS_SUCCESS (5 vs. 0) CUDNN_STATUS_INVALID_VALUE
*** Check failure stack trace: ***
[ubuntu:08202] *** Process received signal ***
[ubuntu:08202] Signal: Aborted (6)
[ubuntu:08202] Signal code: (-6)
[ubuntu:08202] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7ff597ba5330]
[ubuntu:08202] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7ff597802c37]
[ubuntu:08202] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7ff597806028]
[ubuntu:08202] [ 3] /usr/local/lib/libglog.so.0(+0xe799) [0x7ff528486799]
[ubuntu:08202] [ 4] /usr/local/lib/libglog.so.0(+0xfebd) [0x7ff528487ebd]
[ubuntu:08202] [ 5] /usr/local/lib/libglog.so.0(_ZN6google10LogMessage9SendToLogEv+0x258) [0x7ff528489e08]
[ubuntu:08202] [ 6] /usr/local/lib/libglog.so.0(_ZN6google10LogMessage5FlushEv+0xa3) [0x7ff528487a53]
[ubuntu:08202] [ 7] /usr/local/lib/libglog.so.0(_ZN6google15LogMessageFatalD1Ev+0xe) [0x7ff52848a73e]
[ubuntu:08202] [ 8] /home/jhj7905/person_search-master/caffe/build/lib/libcaffe.so(ZN5caffe21CuDNNConvolutionLayerIfE11Forward_gpuERKSt6vectorIPNS_4BlobIfEESaIS5_EES9+0x45e) [0x7ff529aee31e]
[ubuntu:08202] [ 9] /home/jhj7905/person_search-master/caffe/build/lib/libcaffe.so(_ZN5caffe3NetIfE13ForwardFromToEii+0x135) [0x7ff529aad385]
[ubuntu:08202] [10] /home/jhj7905/person_search-master/tools/../caffe/python/caffe/_caffe.so(ZN5boost6python7objects23caller_py_function_implINS0_6detail6callerIMN5caffe3NetIfEEFfiiENS0_21default_call_policiesENS_3mpl7vector4IfRS7_iiEEEEEclEP7_objectSI+0xf9) [0x7ff529f43d49]
[ubuntu:08202] [11] /usr/local/lib/libboost_python.so.1.59.0(ZNK5boost6python7objects8function4callEP7_objectS4+0xca) [0x7ff5290adfaa]
[ubuntu:08202] [12] /usr/local/lib/libboost_python.so.1.59.0(+0x29318) [0x7ff5290ae318]
[ubuntu:08202] [13] /usr/local/lib/libboost_python.so.1.59.0(_ZN5boost6python21handle_exception_implENS_9function0IvEE+0x53) [0x7ff5290b8623]
[ubuntu:08202] [14] /usr/local/lib/libboost_python.so.1.59.0(+0x27c23) [0x7ff5290acc23]
[ubuntu:08202] [15] python(PyEval_EvalFrameEx+0x98d) [0x5244dd]
[ubuntu:08202] [16] python() [0x568b3a]
[ubuntu:08202] [17] python(PyEval_EvalFrameEx+0x2167) [0x525cb7]
[ubuntu:08202] [18] python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[ubuntu:08202] [19] python(PyEval_EvalFrameEx+0x7e8) [0x524338]
[ubuntu:08202] [20] python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[ubuntu:08202] [21] python(PyEval_EvalFrameEx+0x7e8) [0x524338]
[ubuntu:08202] [22] python(PyEval_EvalFrameEx+0xc9a) [0x5247ea]
[ubuntu:08202] [23] python() [0x567d14]
[ubuntu:08202] [24] python(PyRun_FileExFlags+0x92) [0x465bf4]
[ubuntu:08202] [25] python(PyRun_SimpleFileExFlags+0x2ee) [0x46612d]
[ubuntu:08202] [26] python(Py_Main+0xb5e) [0x466d92]
[ubuntu:08202] [27] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7ff5977edf45]
[ubuntu:08202] [28] python() [0x577c2e]
[ubuntu:08202] *** End of error message ***
Aborted (core dumped)

[Cysu]

@jhj7905 I have checked the cudnn v5.1 manual and there was no CUDNN_STATUS_INVALID_VALUE error type relavent to convolution operations.

Could you please check the output of the following command:

ldd caffe/build/install/bin/caffe | grep cudnn

Also note that cudnn v5.1 has cuda-7.5 and cuda-8.0 versions. Please make sure the correct version is installed and linked.

[jhj7905]

@Cysu there is no such file or direcotry when i used ldd caffe/build/install/bin/caffe | grep cudnn...

[Cysu]

@jhj7905 How did you compile the caffe? Did you follow the cmake commands listed in our README?

[jhj7905]

@Cysu Can you show me your cmakelist when you compiled the caffe...

[jhj7905]

@Cysu I recompiled the caffe by modifing the cmakelists.
there is libcudnn.so.5 when i used ldd caffe/build/install/bin/caffe | grep cudnn...
i want to confirm your cmakelists..please show your one???

[Cysu]

@jhj7905 I didn't modify the CMakeLists.txt. Usually it is configured through command line parameters, like the one we shown in the README:

cmake .. -DUSE_MPI=ON -DCUDNN_INCLUDE=/path/to/cudnn/include -DCUDNN_LIBRARY=/path/to/cudnn/lib64/libcudnn.so

Here what I mean is that please make sure the cudnn it linked to is for cuda-8.0, not cuda-7.5.
There are two versions of cudnn-5.1 on the nvidia website for download.

[jhj7905]

@Cysu Do you mean that It is not correct (there is libcudnn.so.5 when i used ldd caffe/build/install/bin/caffe | grep cudnn...)?

i installed cuda-8.0, cudnn-5.1...but still same result like below
cudnn_cov_lay.cu:33] check failed: status == CUDNN_STATUS_SUCCESS( 5 vs. 0 ) CUDN_STATUS_INVALID_VALUE

Could you tell me how to solve the problem in details?

[Cysu]

The libcudnn.so.5 normally links to libcudnn.so.5.1.10. Could you please check the file size of the libcudnn.so.5.1.10? It should be 84163560 bytes. If not, the version is probably not correct.

[jhj7905]

@Cysu I checked the file size of libcudnn.so.5.1.10... like below
-rwxr-xr-x 1 root root 84163560 Aug 30 09:32 libcudnn.so*
-rwxr-xr-x 1 root root 84163560 Aug 30 09:32 libcudnn.so.5*
-rwxr-xr-x 1 root root 84163560 Aug 30 09:32 libcudnn.so.5.1.10*
-rw-r--r-- 1 root root 70364814 Aug 30 09:32 libcudnn_static.a
hmmm...

[Cysu]

Alright, that is correct. I wonder if it is due to out of memory? Could you please check the memory consumption with nvidia-smi?

[jhj7905]

@Cysu I checked the memory consumption like below
| NVIDIA-SMI 375.74 Driver Version: 375.74 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX TIT... On | 0000:02:00.0 On | N/A |
| 32% 75C P2 164W / 250W | 8908MiB / 12206MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX TIT... On | 0000:03:00.0 Off | N/A |
| 22% 40C P8 15W / 250W | 3MiB / 12207MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX TIT... On | 0000:82:00.0 Off | N/A |
| 22% 35C P8 15W / 250W | 3MiB / 12207MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX TIT... On | 0000:83:00.0 Off | N/A |
| 22% 38C P8 15W / 250W | 3MiB / 12207MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Do you think that this is the problem?
I think this is not the problem
If this is the problem, then how to solve the problem?.. and how to use the multi-gpu??
i have a one more question. Could you tell me how to install the openmpi in details

[jhj7905]

cuz..i did not install openmpi... because of python termination

[Cysu]

@jhj7905 It seems that the GPU 0 is almost occupied (9GB / 12GB). You may try to set --gpu 1 and see if it works.

To install openmpi, please download the source from here, then

tar xf openmpi-1.10.7.tar.gz
cd openmpi-1.10.7
./configure --with-cuda=/usr/local/cuda --enable-mpi-thread-multiple
make -j8
sudo make install
cd -

This will by default install it to /usr/local/. To use it, please add the following line to your ~/.bashrc

export PATH=/usr/local/bin:$PATH

Restart the terminal, remove the caffe/build directory, and compile it again with -DUSE_MPI=ON option in cmake.

[jhj7905]

@Cysu I did it as you told me..set --gpu 1..but still did not work......
hmm.....what is your pc spec??...
i have questions about your network.
you use a softmax classifier whether the region is person or not right?
and.. usually the output of bbox is 4..but you have 8 bbox..why? and what does it consist?(for example, x1,x2?? ) can you tell me in details?

[jhj7905]

@Cysu i have a problem..when i build with mpi like below
hmm

jhj7905@ubuntu:~/person_search-master$ python tools/demo.py
[ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_paffinity_hwloc: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_carto_file: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[ubuntu:37345] mca: base: component_find: unable to open /usr/lib/openmpi/lib/openmpi/mca_shmem_posix: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
[ubuntu:37345] *** Process received signal ***
[ubuntu:37345] Signal: Segmentation fault (11)
[ubuntu:37345] Signal code: Address not mapped (1)
[ubuntu:37345] Failing at address: (nil)
[ubuntu:37345] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f1c08b04330]
[ubuntu:37345] [ 1] /usr/local/lib/libopen-pal.so.13(+0x46dcb) [0x7f1be200bdcb]
[ubuntu:37345] *** End of error message ***
Segmentation fault (core dumped)

[Cysu]

@jhj7905 Oh, I forgot to mention that you may need to also add the following line to ~/.bashrc:

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

By the way, could you please also verify that which mpirun refers to /usr/local/bin/mpirun and mpirun --version report 1.10.7?

[Cysu]

@jhj7905 My PC is Titan X Haswell, cuda-8, cudnn-v5.1.

The reason why we have 8 output units for bboxes instead of 4 is that the original py-faster-rcnn implementation was for general object detection, say there could be 20 object classes + 1 background class. Thus they have in total 21 bboxes, one for each class. We inherit this part of code, so there are 2 bboxes for pedestrian and non-pedestrian. The one for non-pedestrian is just for simplicity and has no effect in practice.

[jhj7905]

@Cysu Sincerly thank you for replying my questions.
I understandd the reason why you have 8 output units for bboexs..
anyway.. my statue about openmpi like below
jhj7905@ubuntu:$ which mpirun
/usr/local/bin/mpirun
jhj7905@ubuntu:$ mpirun --version
mpirun: /usr/local/cuda/lib64/libOpenCL.so.1: no version information available (required by /usr/local/lib/libopen-pal.so.13)
mpirun (Open MPI) 1.10.7

Report bugs to http://www.open-mpi.org/community/help/

above case, when i type 'from mpi4py import MPI', then error occur

one more thing. could u tell me how to use the multi-gpu

i should run it by using gpu...cuz you have given me a lot of support..
thank u

[Cysu]

@jhj7905 You mean it's fine to run the demo with one GPU now? That's great.

If you haven't installed the mpi4py package before, you can install it with

pip install mpi4py

The demo is not for multi-gpu. We currently only have the evaluation script supporting multi-gpu. Sorry about the inconvenience.

[jhj7905]

@Cysu I have a question about building the mpi..
do u use opencl when build it?

[Cysu]

@jhj7905 No, I didn't use opencl when building mpi. I used exactly the same commands as I listed above.

[liuajian]

@jhj7905
unbuntu 16.04 + cuda8.0+cudnn5.1就可以了，
openmpi不用安装 系统自带的就是openmpi-1.10.2 直接按作者的cmake .. -DUSE_MPI=ON -DCUDNN_INCLUDE=/path/to/cudnn/include -DCUDNN_LIBRARY=/path/to/cudnn/lib64/libcudnn.so
就可以了

[XinshaoWang]

@liuajian How do you make it?
I have installed all the dependencies as the author list. My work environment is Ubuntu 16.04+cuda8.0.
When I used the cudnn v6.0, the errors are as follows:

-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- system
-- thread
-- chrono
-- date_time
-- atomic
-- Found GFlags: /usr/include
-- Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
-- Found Glog: /usr/include
-- Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found PROTOBUF Compiler: /usr/bin/protoc
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.8.16")
-- Found LMDB: /usr/include
-- Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
-- Found LevelDB: /usr/include
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
-- Found Snappy: /usr/include
-- Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so)
-- CUDA detected: 8.0
-- Found cuDNN (include: /usr/local/cuda-8.0/include, library: /usr/local/cuda-8.0/lib64/libcudnn.so)
-- Added CUDA NVCC flags for: sm_61
-- OpenCV found (/usr/local/share/OpenCV)
-- Found Atlas: /usr/include
-- Found Atlas (include: /usr/include, library: /usr/lib/libatlas.so)
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7")
-- Found NumPy: /usr/lib/python2.7/dist-packages/numpy/core/include (found suitable version "1.11.0", minimum required is "1.7.1")
-- NumPy ver. 1.11.0 found (include: /usr/lib/python2.7/dist-packages/numpy/core/include)
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- python
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Found MPI_C: /usr/local/lib/libmpi.so
-- Found MPI_CXX: /usr/local/lib/libmpi_cxx.so;/usr/local/lib/libmpi.so
-- Found Git: /usr/bin/git (found version "2.7.4")

-- ******************* Caffe Configuration Summary *******************
-- General:
-- Version : (Caffe doesn't declare its version in headers)
-- Git : v0.9999-1625-gaed3884-dirty
-- System : Linux
-- C++ compiler : /usr/bin/c++
-- Release CXX flags : -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
-- Debug CXX flags : -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
-- Build type : Release

-- BUILD_SHARED_LIBS : ON
-- BUILD_python : ON
-- BUILD_matlab : OFF
-- BUILD_docs : ON
-- CPU_ONLY : OFF

-- Dependencies:
-- BLAS : Yes (Atlas)
-- Boost : Yes (ver. 1.58)
-- glog : Yes
-- gflags : Yes
-- protobuf : Yes (ver. 2.6.1)
-- lmdb : Yes (ver. 0.9.17)
-- Snappy : Yes (ver. 1.1.3)
-- LevelDB : Yes (ver. 1.18)
-- OpenCV : Yes (ver. 3.1.0)
-- CUDA : Yes (ver. 8.0)

-- NVIDIA CUDA:
-- Target GPU(s) : Auto
-- GPU arch(s) : sm_61
-- cuDNN : Yes

-- Python:
-- Interpreter : /usr/bin/python2.7 (ver. 2.7.12)
-- Libraries : /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12)
-- NumPy : /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)

-- Documentaion:
-- Doxygen : No
-- config_file :

-- Install:
-- Install path : /home/amos/person_search/caffe/build/install

-- Configuring done
-- Generating done
-- Build files have been written to: /home/amos/person_search/caffe/build
[ 0%] Running C++/Python protocol buffer compiler on /home/amos/person_search/caffe/src/caffe/proto/caffe.proto
Scanning dependencies of target proto
[ 1%] Building CXX object src/caffe/CMakeFiles/proto.dir///include/caffe/proto/caffe.pb.cc.o
[ 1%] Linking CXX static library ../../lib/libproto.a
[ 1%] Built target proto
[ 2%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o
[ 2%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o
[ 4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000312_00000000-5_im2col.cpp4.ii".
CMake Error at cuda_compile_generated_im2col.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_im2col.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:70: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000031f_00000000-5_labeled_matching_layer.cpp4.ii".
CMake Error at cuda_compile_generated_labeled_matching_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_labeled_matching_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:91: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o] Error 1
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000339_00000000-5_absval_layer.cpp4.ii".
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

CMake Error at cuda_compile_generated_absval_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_absval_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:77: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o] Error 1
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000033e_00000000-5_pooling_layer.cpp4.ii".
1 error detected in the compilation of "/tmp/tmpxft_0000034a_00000000-5_cudnn_conv_layer.cpp4.ii".
CMake Error at cuda_compile_generated_pooling_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_pooling_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:105: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o] Error 1
CMake Error at cuda_compile_generated_cudnn_conv_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_conv_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:98: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o] Error 1
1 error detected in the compilation of "/tmp/tmpxft_00000352_00000000-5_concat_layer.cpp4.ii".
CMake Error at cuda_compile_generated_concat_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_concat_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:84: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o] Error 1
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000331_00000000-5_math_functions.cpp4.ii".
CMake Error at cuda_compile_generated_math_functions.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_math_functions.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:63: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o] Error 1
1 error detected in the compilation of "/tmp/tmpxft_0000035a_00000000-5_cudnn_softmax_layer.cpp4.ii".
CMake Error at cuda_compile_generated_cudnn_softmax_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_softmax_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:427: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o] Error 1
CMakeFiles/Makefile2:240: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

[XinshaoWang]

@liuajian @Cysu
When I used cudnn v5.1, the errors are as follows:
~/person_search/caffe/build$ cmake .. -DUSE_MPI=ON -DCUDNN_INCLUDE=/home/amos/cudnn-8.0-linux-x64-v5.1/include -DCUDNN_LIBRARY=/home/amos/cudnn-8.0-linux-x64-v5.1/lib64/libcudnn.so
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- system
-- thread
-- chrono
-- date_time
-- atomic
-- Found GFlags: /usr/include
-- Found gflags (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libgflags.so)
-- Found Glog: /usr/include
-- Found glog (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libglog.so)
-- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so
-- Found PROTOBUF Compiler: /usr/bin/protoc
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5_hl.so;/usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.8.16")
-- Found LMDB: /usr/include
-- Found lmdb (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/liblmdb.so)
-- Found LevelDB: /usr/include
-- Found LevelDB (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libleveldb.so)
-- Found Snappy: /usr/include
-- Found Snappy (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libsnappy.so)
-- CUDA detected: 8.0
-- Found cuDNN (include: /home/amos/cudnn-8.0-linux-x64-v5.1/include, library: /home/amos/cudnn-8.0-linux-x64-v5.1/lib64/libcudnn.so)
-- Added CUDA NVCC flags for: sm_61
-- OpenCV found (/usr/local/share/OpenCV)
-- Found Atlas: /usr/include
-- Found Atlas (include: /usr/include, library: /usr/lib/libatlas.so)
-- Found PythonInterp: /usr/bin/python2.7 (found suitable version "2.7.12", minimum required is "2.7")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7")
-- Found NumPy: /usr/lib/python2.7/dist-packages/numpy/core/include (found suitable version "1.11.0", minimum required is "1.7.1")
-- NumPy ver. 1.11.0 found (include: /usr/lib/python2.7/dist-packages/numpy/core/include)
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- python
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Found MPI_C: /usr/local/lib/libmpi.so
-- Found MPI_CXX: /usr/local/lib/libmpi_cxx.so;/usr/local/lib/libmpi.so
-- Found Git: /usr/bin/git (found version "2.7.4")

-- ******************* Caffe Configuration Summary *******************
-- General:
-- Version : (Caffe doesn't declare its version in headers)
-- Git : v0.9999-1625-gaed3884-dirty
-- System : Linux
-- C++ compiler : /usr/bin/c++
-- Release CXX flags : -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
-- Debug CXX flags : -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
-- Build type : Release

-- BUILD_SHARED_LIBS : ON
-- BUILD_python : ON
-- BUILD_matlab : OFF
-- BUILD_docs : ON
-- CPU_ONLY : OFF

-- Dependencies:
-- BLAS : Yes (Atlas)
-- Boost : Yes (ver. 1.58)
-- glog : Yes
-- gflags : Yes
-- protobuf : Yes (ver. 2.6.1)
-- lmdb : Yes (ver. 0.9.17)
-- Snappy : Yes (ver. 1.1.3)
-- LevelDB : Yes (ver. 1.18)
-- OpenCV : Yes (ver. 3.1.0)
-- CUDA : Yes (ver. 8.0)

-- NVIDIA CUDA:
-- Target GPU(s) : Auto
-- GPU arch(s) : sm_61
-- cuDNN : Yes

-- Python:
-- Interpreter : /usr/bin/python2.7 (ver. 2.7.12)
-- Libraries : /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12)
-- NumPy : /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)

-- Documentaion:
-- Doxygen : No
-- config_file :

-- Install:
-- Install path : /home/amos/person_search/caffe/build/install

-- Configuring done
-- Generating done
-- Build files have been written to: /home/amos/person_search/caffe/build
amos@Amos:~/person_search/caffe/build$ make -j8 && make install
[ 0%] Running C++/Python protocol buffer compiler on /home/amos/person_search/caffe/src/caffe/proto/caffe.proto
Scanning dependencies of target proto
[ 1%] Building CXX object src/caffe/CMakeFiles/proto.dir///include/caffe/proto/caffe.pb.cc.o
[ 1%] Linking CXX static library ../../lib/libproto.a
[ 1%] Built target proto
[ 4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o
[ 4%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o
[ 5%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000905_00000000-5_im2col.cpp4.ii".
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

CMake Error at cuda_compile_generated_im2col.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_im2col.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:70: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_im2col.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
1 error detected in the compilation of "/tmp/tmpxft_000008fb_00000000-5_labeled_matching_layer.cpp4.ii".
CMake Error at cuda_compile_generated_labeled_matching_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_labeled_matching_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:91: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_labeled_matching_layer.cu.o] Error 1
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_0000090e_00000000-5_concat_layer.cpp4.ii".
CMake Error at cuda_compile_generated_concat_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_concat_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:84: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_concat_layer.cu.o] Error 1
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000927_00000000-5_cudnn_conv_layer.cpp4.ii".
1 error detected in the compilation of "/tmp/tmpxft_00000920_00000000-5_absval_layer.cpp4.ii".
CMake Error at cuda_compile_generated_cudnn_conv_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_conv_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:98: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_conv_layer.cu.o] Error 1
CMake Error at cuda_compile_generated_absval_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_absval_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:77: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_absval_layer.cu.o] Error 1
1 error detected in the compilation of "/tmp/tmpxft_00000930_00000000-5_pooling_layer.cpp4.ii".
CMake Error at cuda_compile_generated_pooling_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_pooling_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:105: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_pooling_layer.cu.o] Error 1
/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

/home/amos/person_search/caffe/include/caffe/util/cudnn.hpp(107): error: too few arguments in function call

1 error detected in the compilation of "/tmp/tmpxft_00000919_00000000-5_math_functions.cpp4.ii".
CMake Error at cuda_compile_generated_math_functions.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_math_functions.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:63: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o] Error 1
1 error detected in the compilation of "/tmp/tmpxft_00000937_00000000-5_cudnn_softmax_layer.cpp4.ii".
CMake Error at cuda_compile_generated_cudnn_softmax_layer.cu.o.cmake:266 (message):
Error generating file
/home/amos/person_search/caffe/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_cudnn_softmax_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:427: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_cudnn_softmax_layer.cu.o] Error 1
CMakeFiles/Makefile2:240: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

[XinshaoWang]

@Cysu @liuajian
I could not build caffe successfully using cudnn v5.1 (I downloaded it and linked it by cmake command) or cudnn v6.0(which I used for py-faster-rcnn, combined in cuda as default cudnn version)

When I build caffe without cuDNN library support by setting USE_CUDNN OFF in the CMakeLists.txt, I build caffe successfully.

Could you please tell me how to build it with cudnn support? Thanks very much.

[Cysu]

Could you please check if there are any cudnn under your cuda root? For example, /usr/local/cuda/include/cudnn.hpp. If so, please remove the cudnn files (both .hpp and .so) and try it again with v5.1.

[XinshaoWang]

@Cysu
Yes, it works now when I removed the cudnn v6.0 files from cuda root. Thanks very much.

I thought I could build with cudnn v5.1 support just by specifying the path in the cmake command. But it seems that the cudnn files in the cuda root would influencing the building.

[Cysu]

@XinshaoWang Great to know that!