Reorganize Swig interface for Python about somoclu

[peterwittek]

This is now almost done. The setup.py has also been modified, now the CUDA version should build directly with setuptools under all architectures (66b83f8). Linux with GCC 5.3 and Clang 3.7 were tested, along with NVCC 7.5. @xgdgsc, could you look into whether works on OS X and Windows? Thanks.

[xgdgsc]

It doesn' t work in windows because customize_compiler_for_nvcc only works in Unix (AttributeError: MSVCCompiler instance has no attribute 'compiler_so' in windows). I remember I' ve tried with this approach, but I didn' t find a customize_compiler_for_nvcc written for windows. I may try to implement one later. I have no access to a Mac with Nvidia GPU.

[xgdgsc]

https://github.com/cudamat/cudamat/pull/42/files seems a way worth trying.

[peterwittek]

I got a MacBook Pro, I made some OS X specific changes, but I still cannot compile the Python version. I followed your instructions for installing gcc, but it did not work. If you confirm that it works on OS X, that is fine.

In the meantime, I am trying to set up a developer environment on Windows with a GPU machine.

[peterwittek]

Nothing works in Windows. For the Windows build, the setup script should be rewritten in the previous way.

[xgdgsc]

What' s the error with OS X?

I haven' t succeeded with the approach mentioned above for Windows.

[peterwittek]

The OS X issue is complex. I installed GCC proper and set the environment variables, but setuputils assumed it was clang, so it used a flag that only clang recognizes. It is pointless to fight it.

For the Windows version, let's just revert back to the variant in release 1.5.1.

[xgdgsc]

I tried on a Mac without NVIDIA GPU, seems working for me:

$ make python
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C src python
mkdir -p ./Python/somoclu/src
mkdir -p ./Python/somoclu/src/Windows
cp ../LICENSE ./Python
cp ../Makefile.in ./Python/somoclu/
cp ../configure ./Python/somoclu/
cp ../config.h.in ./Python/somoclu/
cp ../install-sh ./Python/somoclu/
cp ./Makefile.in ./Python/somoclu/src/
cp ./*.h ./Python/somoclu/src/
cp ./mapDistanceFunctions.cpp ./Python/somoclu/src/
cp ./training.cpp ./Python/somoclu/src/
cp ./uMatrix.cpp ./Python/somoclu/src/
cp ./denseCpuKernels.cpp ./Python/somoclu/src/
cp ./denseGpuKernels.cu ./Python/somoclu/src/
cp ./sparseCpuKernels.cpp ./Python/somoclu/src/
cp ./*.cu ./Python/somoclu/src/
swig -c++ -python ./Python/somoclu/somoclu.i
cd Python; python setup.py build
Proceeding without CUDA
running build
running build_py
copying somoclu/somoclu_wrap.py -> build/lib.macosx-10.11-x86_64-2.7/somoclu
running build_ext
building '_somoclu_wrap' extension
/usr/local/bin/gcc-5 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/lib/python2.7/site-packages/numpy/core/include -Isrc -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c somoclu/src/denseCpuKernels.cpp -o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/denseCpuKernels.o -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
somoclu/src/denseCpuKernels.cpp: In function '<built-in>':
somoclu/src/denseCpuKernels.cpp:99:40: warning: '*((void*)& p1 +4)' may be used uninitialized in this function [-Wmaybe-uninitialized]
                 bmus[2 * n + 1] = p1[1];
                                        ^
somoclu/src/denseCpuKernels.cpp:80:18: note: '*((void*)& p1 +4)' was declared here
     unsigned int p1[2] = {0, 0};
                  ^
somoclu/src/denseCpuKernels.cpp:98:36: warning: 'p1' may be used uninitialized in this function [-Wmaybe-uninitialized]
                 bmus[2 * n] = p1[0];
                                    ^
somoclu/src/denseCpuKernels.cpp:80:18: note: 'p1' was declared here
     unsigned int p1[2] = {0, 0};
                  ^
/usr/local/bin/gcc-5 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/lib/python2.7/site-packages/numpy/core/include -Isrc -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c somoclu/src/sparseCpuKernels.cpp -o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/sparseCpuKernels.o -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
somoclu/src/sparseCpuKernels.cpp: In function '<built-in>':
somoclu/src/sparseCpuKernels.cpp:107:40: warning: '*((void*)& p1 +4)' may be used uninitialized in this function [-Wmaybe-uninitialized]
                 bmus[2 * n + 1] = p1[1];
                                        ^
somoclu/src/sparseCpuKernels.cpp:88:9: note: '*((void*)& p1 +4)' was declared here
     int p1[2] = {0, 0};
         ^
somoclu/src/sparseCpuKernels.cpp:106:36: warning: 'p1' may be used uninitialized in this function [-Wmaybe-uninitialized]
                 bmus[2 * n] = p1[0];
                                    ^
somoclu/src/sparseCpuKernels.cpp:88:9: note: 'p1' was declared here
     int p1[2] = {0, 0};
         ^
/usr/local/bin/gcc-5 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/lib/python2.7/site-packages/numpy/core/include -Isrc -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c somoclu/src/mapDistanceFunctions.cpp -o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/mapDistanceFunctions.o -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/usr/local/bin/gcc-5 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/lib/python2.7/site-packages/numpy/core/include -Isrc -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c somoclu/src/training.cpp -o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/training.o -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
somoclu/src/training.cpp: In function 'void train(int, float*, svm_node**, float*, int*, float*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, std::__cxx11::string, float, float, std::__cxx11::string, unsigned int, std::__cxx11::string, std::__cxx11::string, bool, bool)':
somoclu/src/training.cpp:94:9: warning: unused variable 'nProcs' [-Wunused-variable]
     int nProcs = 1;
         ^
/usr/local/bin/gcc-5 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/lib/python2.7/site-packages/numpy/core/include -Isrc -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c somoclu/src/uMatrix.cpp -o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/uMatrix.o -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/usr/local/bin/gcc-5 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/lib/python2.7/site-packages/numpy/core/include -Isrc -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c somoclu/somoclu_wrap.cxx -o build/temp.macosx-10.11-x86_64-2.7/somoclu/somoclu_wrap.o -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
/usr/local/bin/g++-5 -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-2.7/somoclu/src/denseCpuKernels.o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/sparseCpuKernels.o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/mapDistanceFunctions.o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/training.o build/temp.macosx-10.11-x86_64-2.7/somoclu/src/uMatrix.o build/temp.macosx-10.11-x86_64-2.7/somoclu/somoclu_wrap.o -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/sqlite/lib -lgomp -o build/lib.macosx-10.11-x86_64-2.7/_somoclu_wrap.so

[peterwittek]

Yes, that is right. Thanks.

OS X is so clever that the CUDA libdir is sometimes $CUDAHOME/lib even on a 64-bit platform. I made some changes to make CUDA detection more robust on such retarded systems. Otherwise, the Windows CUDA compilation stuff was moved back from v1.5.1.

If you agree, we can close this, and proceed with the new release.

[xgdgsc]

I agree we can close this and proceed with the new release now. We can wait for or participate in cudamat/cudamat#53 which might leads to a general solution later.