pip install error on OS X 10.11 about falconn

[ludwigschmidt]

Hi, thanks for your feedback! I haven't tested the Python 3 version myself, maybe @ilyaraz has more insights. Both of us have a paper deadline this week, so it might take us a few days until we can look into this issue in more detail. Sorry about that!

[w4nderlust]

I actually tried to download it and compile it myself, but I'm experiencing compilation problems too. If you think it can be useful I can post those errors too.

[ludwigschmidt]

Yes, that would probably be useful. Thanks!

[w4nderlust]

Ok, first of all I had to change the include dir inside the make file to /Users/w4nderlust/Development/ann-eval/venv/lib/python3.5/site-packages/numpy/core/include, where I installed bumpy inside the virtual environment, otherwise I was getting bumpy inclusion errors when running make python_swig. So I can run make python_swig with just few warnings. I then run make python_package and make python_package_install.
After installing the package in the virtual environment I try to runt the tests with make run_all_python_tests and I get:

py.test src/python/test/wrapper_test.py
============================== test session starts ==============================
platform darwin -- Python 3.5.1 -- py-1.4.31 -- pytest-2.6.4
collected 0 items / 1 errors 

==================================== ERRORS =====================================
_______________ ERROR collecting src/python/test/wrapper_test.py ________________
/usr/local/lib/python3.5/site-packages/py/_path/local.py:650: in pyimport
    __import__(modname)
<frozen importlib._bootstrap>:969: in _find_and_load
    ???
<frozen importlib._bootstrap>:954: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:892: in _find_spec
    ???
<frozen importlib._bootstrap>:873: in _find_spec_legacy
    ???
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:137: in find_module
    source_stat, co = _rewrite_test(state, fn_pypath)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:272: in _rewrite_test
    rewrite_asserts(tree)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:330: in rewrite_asserts
    AssertionRewriter().run(mod)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:539: in run
    new.extend(self.visit(child))
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ast.py:245: in visit
    return visitor(node)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:647: in visit_Assert
    top_condition, explanation = self.visit(assert_.test)
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ast.py:245: in visit
    return visitor(node)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:778: in visit_Compare
    left_res, left_expl = self.visit(comp.left)
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ast.py:245: in visit
    return visitor(node)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:769: in visit_Attribute
    value, value_expl = self.visit(attr.value)
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ast.py:245: in visit
    return visitor(node)
/usr/local/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:682: in visit_Name
    locs = ast.Call(self.builtin("locals"), [], [], None, None)
E   TypeError: Call constructor takes either 0 or 3 positional arguments
============================ 1 error in 0.15 seconds ============================
make: *** [run_all_python_tests] Error 1

[ludwigschmidt]

These are different errors than in your initial gist, correct?

To me it looks like the swig wrapper is not working correctly. Does FALCONN work with Python 2 on your machine?

[w4nderlust]

Yes, these are errors while trying to compile from sources, the ones in the gist were errors when trying to install with pip. I just tried to install FALCONN on python 2 with pip and it installed just fine.

[rjchacko]

I'm now running into this issue using pip with Python 3.5 on Debian, (this docker image actually). I'm curious if a workaround was found already? I tried adding a symlink to the numpy include library but that didn't help.

[ilyaraz]

@w4nderlust @rjchacko can you try to clone the repo and run make python_package_install? (You need gcc, swig and numpy for this to possibly succeed.)

[w4nderlust]

Now I'm on OSX 10.12. Installed swig using homebrew, so swig 3.0.0 is installed as swig not swig3.0. I changed that in the Makefile and also changed this CXX=/usr/local/Cellar/gcc/5.3.0/bin/g++-5 as I installed gcc through homebrew too.

The problem is that i get a clang error when i try to compile even if there's no clang in the makefil. It could be that somewhere else gcc is called (and not the path that I pasted) and that is defaulted to clang.

The command failing is:
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ifalconn/src/include -Ifalconn/external/eigen -I/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c falconn/swig/falconn_wrap.cc -o build/temp.macosx-10.6-x86_64-2.7/falconn/swig/falconn_wrap.o -std=c++11 -march=native -O3

Attached is the full output.
output.txt

[ilyaraz]

@w4nderlust Perhaps, this is called during python setup.py build. Thanks, I will look into it!

[ilyaraz]

@ludwigschmidt Since I don't have a machine with OSX, can you try to reproduce it? I suspect the problem is with mixing libc++ and libstdc++.

[ludwigschmidt]

Just to confirm: the issue of @w4nderlust occurs with Python 2.7, right? I can run make python_package_install and pip install falconn in a fresh Python 2.7 virtualenv without any issues. I'm using OS X 10.11, so version 10.11 vs 10.12 could be a difference.

I'm also not 100% sure why clang is compiling the code in the end. I guess the pip machinery uses the system's default compiler?

What is your compiler, @w4nderlust ? I get the following output for clang -v:

Apple LLVM version 7.3.0 (clang-703.0.31)
Target: x86_64-apple-darwin15.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[ludwigschmidt]

Regarding the questions about Python 3.5 (@w4nderlust's original gist and maybe also the issue of @rjchacko): I ran into the same problem locally. It looks like this is a bug in swig and fixed in version 3.0.8:

https://sourceforge.net/p/swig/mailman/message/34688087/

After I upgraded to swig version 3.0.8 I could compile FALCONN again.

Let us know if that also solves the problem for you.

[w4nderlust]

@ludwigschmidt yes, python 2.7. My clang version is 8:

Apple LLVM version 8.0.0 (clang-800.0.38) Target: x86_64-apple-darwin16.0.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Anyway, i created a new venv with python 2.7, installed numpy and then run pip install falconn and had no issue.

I created a new virtual environment with python3, installed numpy and tried to install falconn and got the error attached.

falconn_p3_error.txt

I also tried to compile in that venv, and got this:

falcon_pr_error_from_source.txt

My swig, installed from homebrew, is version 3.0.8 and was compiled with clang (maybe that's the problem?)

[ludwigschmidt]

OK, glad to hear that pip install falconn now works with Python 2.7. Do you know what changed since the last try?

Regarding Python 3, the second log does not contain any errors, correct? So it looks like you can compile the code from the repo but not from PyPi. Probably the reason is that the PyPi package contains a swig wrapper generated with a version of swig that contained the bug (pre 3.0.8).

@ilyaraz , can you create a new PyPi package generated with a current swig (3.0.8)? Then we should have addressed all points in this issue :-) Thanks!

[w4nderlust]

@ludwigschmidt last time i was trying to compile from source, while this time I downloaded the package from pip. In the meantime I also updated swig, so it is likely that it was the problem for python 2.7 too. Sorry if I can't be more precise, but I continuously have to keep up with latest versions for work and I don't always keep track of all the changes that I make to my configuration.
And yes, in the last output there are a few warning but no error. So I guess again that updating to swig solved the problem for python 3 too. Thanks for your effort.

[ludwigschmidt]

Cool, glad that FALCONN works for you with both Python 2 and 3 now! We'll update the PyPi package soon.

[ilyaraz]

So, the only problem with the PyPi package is that it was built with a wrong version of swig?

[ludwigschmidt]

Yes, that is my understanding.

[ludwigschmidt]

It's somewhat unfortunate that swig caused this bug. I guess in an ideal world, we'd have our own Python wrapping code independent of swig. But for now swig is what we have.

[ilyaraz]

@ludwigschmidt can you try this version of the package?

falconn.tar.gz

[ludwigschmidt]

python setup.py build works in a Python 3.5.1. virtualenv on my Mac OS X 10.11 installation. Thanks!

[ilyaraz]

I've made a new release and uploaded the new package to PyPi.