uniomni / pypar Goto Github PK
View Code? Open in Web Editor NEWAutomatically exported from code.google.com/p/pypar
Automatically exported from code.google.com/p/pypar
Hi, I have compiled Pypar on windows 32 s.o.
When I try to launch the Paypar test i can't find the libmpi.so:
mpi = CDLL('libmpi.so.0', RTLD_GLOBAL)
I use MPICH2.
Any suggestion?
Original issue reported on code.google.com by [email protected]
on 12 Jan 2012 at 9:28
Running 'python setup.py install' on OS X with MPICH2 does not set up pypar
correctly, although the 'setup.py' script seems to run without any errors.
The following output is generated:
romesh: source > python setup.py install
setup.py:34: DeprecationWarning: The popen2 module is deprecated. Use the
subprocess module.
import popen2 # FIXME: Replace with subprocess -
http://docs.python.org/library/subprocess.html#replacing-older-functions-with-th
e-subprocess-module
/Users/romesh/mpich2-install/bin/mpicc: line 248: echo: write error: Broken
pipe
/Users/romesh/mpich2-install/bin/mpicc: line 248: echo: write error: Broken
pipe
/Users/romesh/mpich2-install/bin/mpicc: line 248: echo: write error: Broken
pipe
+ '[' no = no ']'
+ unset 'allargs[0]'
++ expr 0 + 1
+ argno=1
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ linking=no
+ '[' yes = no ']'
++ expr 1 + 1
+ argno=2
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ '[' yes = no ']'
++ expr 2 + 1
+ argno=3
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ '[' yes = no ']'
++ expr 3 + 1
+ argno=4
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ '[' yes = no ']'
++ expr 4 + 1
+ argno=5
+ '[' 5 -eq 0 ']'
+ '[' yes = yes ']'
+ mpilibs='-lpmpich -lmpich -lopa'
+ CFLAGS=
+ LDFLAGS=' '
+ '[' -n '' ']'
+ '[' no = yes ']'
+ gcc -c /var/folders/eW/eWXFeb8cGoKDvOMgR+R4Q++++TI/-Tmp-/tmp7YHukX.c -o
/var/folders/eW/eWXFeb8cGoKDvOMgR+R4Q++++TI/-Tmp-/tmp7YHukX.o
-I/Users/romesh/mpich2-install/include
+ rc=0
+ exit 0
gcc -I/Users/romesh/mpich2-install/include
-L/Users/romesh/mpich2-install/lib -L/Users/romesh/mpich2-install/lib
-lpmpich -lmpich -lopa -lpthread
+ '[' no = no ']'
+ unset 'allargs[0]'
++ expr 0 + 1
+ argno=1
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ linking=no
+ '[' yes = no ']'
++ expr 1 + 1
+ argno=2
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ '[' yes = no ']'
++ expr 2 + 1
+ argno=3
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ '[' yes = no ']'
++ expr 3 + 1
+ argno=4
+ for arg in '"$@"'
+ addarg=yes
+ case "$arg" in
+ '[' yes = no ']'
++ expr 4 + 1
+ argno=5
+ '[' 5 -eq 0 ']'
+ '[' yes = yes ']'
+ mpilibs='-lpmpich -lmpich -lopa'
+ CFLAGS=
+ LDFLAGS=' '
+ '[' -n '' ']'
+ '[' no = yes ']'
+ gcc -c /var/folders/eW/eWXFeb8cGoKDvOMgR+R4Q++++TI/-Tmp-/tmp7YHukX.c -o
/var/folders/eW/eWXFeb8cGoKDvOMgR+R4Q++++TI/-Tmp-/tmp7YHukX.o
-I/Users/romesh/mpich2-install/include
+ rc=0
+ exit 0
running install
running build
running build_py
creating build
creating build/lib.macosx-10.6-universal-2.6
creating build/lib.macosx-10.6-universal-2.6/pypar
copying __init__.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying __metadata__.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying compile.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying compile_pypar_locally.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying network_timing.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying pypar.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying pypar_balancer.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying test_init.py -> build/lib.macosx-10.6-universal-2.6/pypar
copying test_pypar.py -> build/lib.macosx-10.6-universal-2.6/pypar
running build_ext
building 'pypar.mpiext' extension
creating build/temp.macosx-10.6-universal-2.6
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os
-Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch ppc -arch x86_64
-pipe -I/Users/romesh/mpich2-install/include
-I/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/num
py/core/include
-I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c
mpiext.c -o build/temp.macosx-10.6-universal-2.6/mpiext.o
mpiext.c: In function ‘type_map’:
mpiext.c:124: warning: cast from pointer to integer of different size
mpiext.c: In function ‘op_map’:
mpiext.c:165: warning: cast from pointer to integer of different size
gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -arch i386 -arch ppc
-arch x86_64 build/temp.macosx-10.6-universal-2.6/mpiext.o
-L/Users/romesh/mpich2-install/lib -lpmpich -lmpich -lopa -lpthread -o
build/lib.macosx-10.6-universal-2.6/pypar/mpiext.so
ld: warning: in /Users/romesh/mpich2-install/lib/libpmpich.dylib, file is
not of required architecture
ld: warning: in /Users/romesh/mpich2-install/lib/libmpich.dylib, file is
not of required architecture
ld: warning: in /Users/romesh/mpich2-install/lib/libopa.a, file is not of
required architecture
ld: warning: in /Users/romesh/mpich2-install/lib/libpmpich.dylib, file is
not of required architecture
ld: warning: in /Users/romesh/mpich2-install/lib/libmpich.dylib, file is
not of required architecture
ld: warning: in /Users/romesh/mpich2-install/lib/libopa.a, file is not of
required architecture
running install_lib
copying build/lib.macosx-10.6-universal-2.6/pypar/mpiext.so ->
/Library/Python/2.6/site-packages/pypar
running install_egg_info
Removing /Library/Python/2.6/site-packages/Pypar-2.1.4-py2.6.egg-info
Writing /Library/Python/2.6/site-packages/Pypar-2.1.4-py2.6.egg-info
But pypar cannot be imported in python:
romesh: source > python
Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypar
ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
python compile_pypar_manually.py
in the pypar directory, or by using
python setup.py install
WARNING: MPI library could not be initialised - running sequentially
>>>
Original issue reported on code.google.com by [email protected]
on 22 Dec 2009 at 10:47
When attempting to 'import pypar', the following error message is generated
Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypar
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pypar.py", line 863, in <module>
mpi = CDLL('libmpi.so.0', RTLD_GLOBAL)
File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ctypes/_
_init__.py",
line 345, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(libmpi.so.0, 10): image not found
>>>
This problem is occuring due to these lines in pypar.py:
# Work around bug in OpenMPI (December 2009):
# https://bugs.launchpad.net/ubuntu/+source/petsc4py/+bug/232036
from ctypes import *
mpi = CDLL('libmpi.so.0', RTLD_GLOBAL)
The problem is that (at least on OS X machines), 'libmpi.so.0' does not
exist (tested with both MPICH2 and OpenMPI). However, 'libmpi.0.dylib' is
the corresponding library on OS X. So the solution would be to use some
more sophisticated way of choosing which library file to use.
For the record, changing the string to 'libmpi.0.dylib' on OS X fixes the
problem with importing pypar
>>> import pypar
Pypar (version 2.1.4) initialised MPI OK with 1 processors
>>>
Original issue reported on code.google.com by [email protected]
on 22 Dec 2009 at 10:26
It is desirable for pypar to release the GIL when calling blocking routines
such as MPI_recv. This way communication can be made threaded within each MPI
program instance. This is especially problematic for my use case because pypar
has no non-blocking alternative calls.
The following python snippet, though of no practical use, illustrates the issue
when run on two processors.
import time, threading, pypar, random
message = None
def communicator_thread() :
global message
message = pypar.receive(source = pypar.mpiext.MPI_ANY_SOURCE)
if pypar.rank()==0 :
threading.Thread(target=communicator_thread).start()
while message is None :
print "Processing happens"
time.sleep(1)
print "Message is handled"
else :
time.sleep(3.5)
pypar.send("Hello",0)
What actually happens when running this with current version of pypar:
Message is handled
(program exits)
What happens with a GIL release, which I'm claiming is preferable:
Processing happens
Processing happens
Processing happens
Message is handled
(program exits)
What version of the product are you using? On what operating system?
pypar 2.1.4_94; python 2.6; Intel MPI; Linux
Please provide any additional information below.
A simple patch for this issue is as follows:
--- mpiext.c 2009-12-22 07:21:30.000000000 +0000
+++ mpiext.c 2010-10-24 14:17:14.000000000 +0100
@@ -211,8 +211,11 @@ static PyObject *receive_string(PyObject
if (!PyArg_ParseTuple(args, "s#ii", &s, &length, &source, &tag))
return NULL;
+ Py_BEGIN_ALLOW_THREADS;
/* call the MPI routine */
error = MPI_Recv(s, length, MPI_CHAR, source, tag, MPI_COMM_WORLD, &status);
+ Py_END_ALLOW_THREADS;
if (error != 0) {
MPI_Comm_rank(MPI_COMM_WORLD, &myid);
@@ -468,9 +473,10 @@ static PyObject *receive_array(PyObject
if (!mpi_type) return NULL;
/* call the MPI routine */
+ Py_BEGIN_ALLOW_THREADS;
error = MPI_Recv(x->data, count, mpi_type, source, tag,
MPI_COMM_WORLD, &status);
-
+ Py_END_ALLOW_THREADS;
/* Do not DECREF x as it must be returned to Python */
if (error != 0) {
Original issue reported on code.google.com by [email protected]
on 24 Oct 2010 at 4:01
This problem occurs if the python MPI library cannot be initialised in
Python. The error message on the Python console is
Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypar
ERROR: C extension mpiext could not be imported.
Please compile mpiext.c e.g. by running
python compile_pypar_manually.py
in the pypar directory, or by using
python setup.py install
WARNING: MPI library could not be initialised - running sequentially
>>>
However, there is no file 'compile_pypar_manually.py'. This message should
probably be referring to 'compile_pypar_locally.py'
Original issue reported on code.google.com by [email protected]
on 22 Dec 2009 at 10:18
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.