opencv 3.3.0: python 3.5: ImportError: numpy.core.multiarray failed to import about berryconda

[jjhelmus]

I tried to replicate this on a Raspberry Pi 3 by creating a new environment using conda create -n opencv_test python=3.5 opencv. I was able to import cv2 fine.

Can you provide some additional details on the environment where you are seeing this error? How was berryconda installed, what hardware is being used, how was opencv was installed, what is the output of conda list in the environment with opencv.

[alexryan]

Here's the precise history of what I did from my notes:

In an environment using python 3.5.4.

`pi@qacker:~/bitsy_gen3/dev-tools/qacker $ source activate qt

(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ conda list

packages in environment at /home/pi/berryconda3/envs/qt:

ca-certificates 2017.7.27.1 0
certifi 2018.1.18 py35hd55c48c_0
ncurses 5.9 0
openssl 1.0.2l 0
pip 9.0.1 py35_0
python 3.5.4 0
readline 6.3 0
setuptools 38.4.0 py35he1abd54_0
sqlite 3.15.0 0
tk 8.6.6 0
wheel 0.30.0 py35haa2052f_0
xz 5.2.2 0
zlib 1.2.11 0

(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ conda search opencv
Fetching package metadata .......
opencv 3.3.0 py36hb02bd27_0 defaults
3.3.0 py27h0ae2b19_0 defaults
3.3.0 py35hb6a5d43_0 defaults

(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ conda install opencv
Fetching package metadata .......
Solving package specifications: .

Package plan for installation in environment /home/pi/berryconda3/envs/qt:

The following NEW packages will be INSTALLED:

bzip2:       1.0.6-0              
ffmpeg:      3.2.4-h11ee089_0     
giflib:      5.1.4-h3b1de47_0     
hdf5:        1.8.17-0             
jasper:      1.900.1-hb74bc11_4   
jpeg:        9b-0                 
libgfortran: 3.0.0-0              
libpng:      1.6.28-0             
libtiff:     4.0.6-0              
libwebp:     0.5.2-hcb3eeb6_7     
numpy:       1.14.0-py35h0995c57_0
openblas:    0.2.19-0             
opencv:      3.3.0-py35hb6a5d43_0 
x264:        20131217-h754e1d8_0  

Proceed ([y]/n)? y

numpy-1.14.0-p 100% |################################################################################################################################| Time: 0:01:40 77.20 kB/s
numpy-1.14.0-p 100% |################################################################################################################################| Time: 0:00:42 179.93 kB/s
opencv-3.3.0-p 100% |################################################################################################################################| Time: 0:01:08 219.59 kB/s
(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ `

Immediately afterwards I got the the error when I tried to import python within this environment.

The python version defaulted to 3.5.4 when I created the environment by specifying 3.5.

I tried to build opencv from scratch using this install guide

However, cmake gave me an error saying that it wanted python 3.5.3 instead:
-- Could NOT find PythonLibs: Found unsuitable version "3.5.3", but required is exact version "3.5.4" (found /usr/lib/arm-linux-gnueabihf/libpython3.5m.so) Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/pi/berryconda3/envs/qt/lib/python3.5/site-packages/numpy/__init__.py", line 166, in <module> from . import random File "/home/pi/berryconda3/envs/qt/lib/python3.5/site-packages/numpy/random/__init__.py", line 99, in <module> from .mtrand import * ImportError: /home/pi/berryconda3/envs/qt/lib/python3.5/site-packages/numpy/random/mtrand.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: PyFPE_jbuf

I created a new environment with that version:
$ conda create --name gui python=3.5.3
but cmake gave me a similar error

-- Found PythonLibs: /usr/lib/arm-linux-gnueabihf/libpython3.5m.so (found suitable exact version "3.5.3") Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/pi/berryconda3/envs/gui/lib/python3.5/site-packages/numpy/__init__.py", line 166, in <module> from . import random File "/home/pi/berryconda3/envs/gui/lib/python3.5/site-packages/numpy/random/__init__.py", line 99, in <module> from .mtrand import * ImportError: /home/pi/berryconda3/envs/gui/lib/python3.5/site-packages/numpy/random/mtrand.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: PyFPE_jbuf
The only way I was able to get opencv to build was to follow the author's instructions using virtualenv.

I hope this is helpful.

[jjhelmus]

Was there any more of the error message? Can you try running python -c "import numpy.core.multiarray"?

[alexryan]

My apologies, but it looks like this might be my bad.
I had to reinstall opencv in the virtual environment, but that particular import works correctly after doing so.

Python 3.5.4 | packaged by rpi | (default, Aug  9 2017, 08:07:25) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> import numpy.core.multiarray
>>> quit()
(qt) pi@qacker:~ $ python -c "import numpy.core.multiarray"
(qt) pi@qacker:~ $ 

However, I am able to reproduce the error running my shell script.

Digging a little deeper it seems like the problem might be caused by having a different version of numpy in my PATH.

(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ cat qacker-pi.sh 
#!/bin/bash

export PYTHONPATH=".:/home/pi/bitsy_gen3/Shared:/usr/lib/python3/dist-packages:$PYTHONPATH"
export PYTHONPATH="/usr/local/lib/python3.5/dist-packages:$PYTHONPATH"
export PYTHONPATH="/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.3.1-py3.5.egg:$PYTHONPATH"
python qacker-PyQt5.py

(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ python
Python 3.5.4 | packaged by rpi | (default, Aug  9 2017, 08:07:25) 
[GCC 4.9.2] on linux

Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/home/pi/berryconda3/envs/qt/lib/python35.zip', '/home/pi/berryconda3/envs/qt/lib/python3.5', '/home/pi/berryconda3/envs/qt/lib/python3.5/plat-linux', '/home/pi/berryconda3/envs/qt/lib/python3.5/lib-dynload', '/home/pi/berryconda3/envs/qt/lib/python3.5/site-packages']
>>> quit()

(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ export PYTHONPATH=".:/home/pi/bitsy_gen3/Shared:/usr/lib/python3/dist-packages:$PYTHONPATH"
(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ export PYTHONPATH="/usr/local/lib/python3.5/dist-packages:$PYTHONPATH"
(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ export PYTHONPATH="/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.3.1-py3.5.egg:$PYTHONPATH"
(qt) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ python
Python 3.5.4 | packaged by rpi | (default, Aug  9 2017, 08:07:25) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> print(sys.path)
['', '/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.3.1-py3.5.egg', '/usr/local/lib/python3.5/dist-packages', '/home/pi/bitsy_gen3/dev-tools/qacker', '/home/pi/bitsy_gen3/Shared', '/usr/lib/python3/dist-packages', '/home/pi/berryconda3/envs/qt/lib/python35.zip', '/home/pi/berryconda3/envs/qt/lib/python3.5', '/home/pi/berryconda3/envs/qt/lib/python3.5/plat-linux', '/home/pi/berryconda3/envs/qt/lib/python3.5/lib-dynload', '/home/pi/berryconda3/envs/qt/lib/python3.5/site-packages']
>>> import cv2
ImportError: numpy.core.multiarray failed to import
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: numpy.core.multiarray failed to import
>>> 

I was not able to find a version of PyQt5 that could be installed via pip or conda, so I had to install via apt-get.
Similarly I was not able to find a working version of paho-mqtt for python 3.5 on raspbian, so I had to build externally and add to my path.

Interestingly, however, I can set the PYTHONPATH exactly the same in my virtualenv environment (where I built openCV from scratch) without getting this error.
So, It's still a bit of a mystery of why it works in virtualenv and not in berryconda.

(qacker_gui) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ export PYTHONPATH=".:/home/pi/bitsy_gen3/Shared:/usr/lib/python3/dist-packages:$PYTHONPATH"
(qacker_gui) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ export PYTHONPATH="/usr/local/lib/python3.5/dist-packages:$PYTHONPATH"
(qacker_gui) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ export PYTHONPATH="/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.3.1-py3.5.egg:$PYTHONPATH"

(qacker_gui) pi@qacker:~/bitsy_gen3/dev-tools/qacker $ python
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170124] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> print(sys.path)
['', '/usr/local/lib/python3.5/dist-packages/paho_mqtt-1.3.1-py3.5.egg', '/usr/local/lib/python3.5/dist-packages', '/home/pi/bitsy_gen3/dev-tools/qacker', '/home/pi/bitsy_gen3/Shared', '/usr/lib/python3/dist-packages', '/home/pi/.virtualenvs/qacker_gui/lib/python35.zip', '/home/pi/.virtualenvs/qacker_gui/lib/python3.5', '/home/pi/.virtualenvs/qacker_gui/lib/python3.5/plat-arm-linux-gnueabihf', '/home/pi/.virtualenvs/qacker_gui/lib/python3.5/lib-dynload', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-arm-linux-gnueabihf', '/home/pi/.virtualenvs/qacker_gui/lib/python3.5/site-packages']
>>> import cv2
>>> import numpy.core.multiarray
>>> quit()

[jjhelmus]

It is quite possible that the Python extension modules built with the system Python are not compatible with Berryconda. Setting the PYTHONPATH variable to the system dist-/site-packages directory is trying to mix these two sets of extension modules.

I have a suspicion that this may be a result of the different configuration that were used to build Python for Berryconda and Raspian. Specifically Raspian uses the --with-fpectl flag where as Berryconda does not enable this flag. The resulting binaries have different ABIs, see bpo-29137.

My recommendation is to build anything not included by Berryconda from source using the Python provided by Berryconda. I understand that this can be a difficult task but it seem to be the only method to insure binary compatibility.