bit66com / python-multiprocessing Goto Github PK

Hello,
I'm forwarding the Debian bug 559574: http://bugs.debian.org/559574 .

multiprocessing contains a couple of typos, for 'unkown' instead of
'unknown'. In the above link there is also a trivial patch to fix them.

There are chances that these typos are already fixed in the python.org SVN
repo (I didn't check): in this case maybe a new backport would be nice.

Regards,
Sandro

Small script, adding few tuples with objects to the Queue
gives me:

--------------8<-------------
Traceback (most recent call last):
  File
"/home/some_path/src/multiprocessing/multiprocessing-2.6.0.2-py2.4-linux-i686.eg
g/multiprocessing/queues.py",
line 242, in _feed
    send(obj)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed
--------------8<-------------

multiprocessing-2.6.0.2-py2.4-linux-i686.egg (installed via easy_install
under the custom --dest-dir)
Python 2.4.4
debian, kernel 2.6.x

But the same script works fine with Queue.Queue.

I will try to give you more details including src code of a script if required.

There has been a regression from pyprocessing. New functionality was
provided to install the specific type picklers, this was completly broken
according to this bug:

http://bugs.python.org/issue4892

It would be nice if the patch mentioned there was backed out or fixed some
way. Without it sockets can't be pickled which prevents one to have a
server that dispatches requests to subprocesses.

I'm not sure if this should be construed as a bug in the documentation or 
in the implementation itself ... When allocating shared memory using using 
sharedctypes.RawArray (and potentially others), the memory is not always 
initialised to zero. The docs explicitly say that RawArray returns a zeroed 
array.

This seems to only occur when one array is deleted and a new one allocated, 
and only for relatively large arrays. In this case the new array contains 
values from the previous one. I assume this is due to the fact that whilst 
the memory in the heap is initially zeroed, no rezeroing occurs when a 
segment of the heap is reused. 

a simple example which reproduces it in on my machine is as follows:

from multiprocessing import sharedctypes
a = sharedctypes.RawArray('i', 10000)

for i in range(10000):
    a[i] = i

print a[1000] ## 1000
del(a)
b = sharedctypes.RawArray('i', 10000)
print b[1000] ## 1000

I ran the commands manually in the console, which I suspect might be 
necessary to give the python garbage collector enough time to get around to 
cleaning up a before b is allocated.

I'm using multiprocessing 2.6.2.1, python 2.5.2, on ubuntu 8.04 x64

I just noticed that the fix[0] for #4660[1] is not included in 
python-multiprocessing.

[0] http://bugs.python.org/issue4660
[1] http://svn.python.org/view?rev=74327&view=rev

The following line in the Manager.decref() method raises an exception (at
least when logging in enabled):

util.debug('disposing of obj with id %d', ident)

This is because the type of 'ident' is str, not int. The following patch
fixes this problem. (This seems like it may be causing problem with
execution of finalization functions in the slave processes...)

Should we update from the current Python release26-maint branch to bring
in bug fixes from the last 9-10 months?

What steps will reproduce the problem?

1. Install latest MacPorts Python 2.4.6
2. Install setuptools
3. "easy_install multiprocessing"

Here's the console output :

[prompt]$ easy_install multiprocessing
Searching for multiprocessing
Reading http://pypi.python.org/simple/multiprocessing/
Reading http://code.google.com/p/python-multiprocessing
Best match: multiprocessing 2.6.2.1
Downloading 
http://pypi.python.org/packages/source/m/multiprocessing/multiprocessing-2.6.2.1
.tar.gz#md5=5cc484396c040102116ccc2355379c72
Processing multiprocessing-2.6.2.1.tar.gz
Running multiprocessing-2.6.2.1/setup.py -q bdist_egg --dist-dir 
/var/folders/D5/D5kUGxNgGI8-yz5btga6y++++TM/-Tmp-/easy_install-Z_sYwg/multiproce
ssing-2.6.2.1/egg-dist-tmp-5eXS_a
Modules/_multiprocessing/multiprocessing.c: In function 
‘multiprocessing_sendfd’:
Modules/_multiprocessing/multiprocessing.c:100: warning: implicit declaration 
of function ‘CMSG_SPACE’
Modules/_multiprocessing/multiprocessing.c:117: warning: implicit declaration 
of function ‘CMSG_LEN’
In file included from Modules/_multiprocessing/socket_connection.c:202:
Modules/_multiprocessing/connection.h: In function ‘connection_new’:
Modules/_multiprocessing/connection.h:51: warning: format ‘%zd’ expects 
type ‘signed size_t’, but argument 3 has type ‘int’
Modules/_multiprocessing/connection.h: In function ‘connection_repr’:
Modules/_multiprocessing/connection.h:468: warning: format ‘%zd’ expects 
type ‘signed size_t’, but argument 4 has type ‘int’
Modules/mmapmodule.c: In function ‘new_mmap_object’:
Modules/mmapmodule.c:947: warning: implicit declaration of function ‘open’
Modules/mmapmodule.c:947: error: ‘O_RDWR’ undeclared (first use in this 
function)
Modules/mmapmodule.c:947: error: (Each undeclared identifier is reported only 
once
Modules/mmapmodule.c:947: error: for each function it appears in.)
error: Setup script exited with error: command '/usr/bin/gcc-4.2' failed with 
exit status 1
[prompt]$ 

Please ask if you need more information on the installation environment. Note 
that other eggs with C code install correctly.

Thanks by advance

I've implemented a monkey patch that reduces the amount of changes to a
minimum. It adds several properties and function aliases to the threading
module. It also implements a 2.6 style property class.

Comments? Is this step too radical?

What steps will reproduce the problem?
1. make
2. make test --> error!!

I can import multiprocessing fine and use other functions, but sleeping and
self._wlock.release() from queues.py cause a not implemented error.

What is the expected output? 

successful use.

What do you see instead?

$ make test
find Lib/ \( -name '*.py[co]' -or -name '*.so' \) -exec rm {} \;
rm -rf build/sphinx
python2.5 setup.py build_ext -i
running build_ext
copying build/lib.macosx-10.5-i386-2.5/multiprocessing/_multiprocessing.so
-> Lib/multiprocessing
PYTHONPATH="Lib/" python2.5 -tt -c "from multiprocessing.tests import main;
main()"
test_array (multiprocessing.tests.WithProcessesTestArray) ... ok
test_getobj_getlock_obj (multiprocessing.tests.WithProcessesTestArray) ... ok
test_rawarray (multiprocessing.tests.WithProcessesTestArray) ... ok
test_notify (multiprocessing.tests.WithProcessesTestCondition) ...
Exception in thread Thread-9:
Traceback (most recent call last):
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threa
ding.py",
line 486, in __bootstrap_inner
    self.run()
  File
"/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threa
ding.py",
line 446, in run
    self.__target(*self.__args, **self.__kwargs)
  File
"/Users/lucy/ExtProjects/python-multiprocessing/Lib/multiprocessing/tests.py",
line 613, in f
    sleeping.release()
OSError: [Errno 78] Function not implemented


What version of the product are you using? On what operating system?

2.6.1.1 r61

mac os 10.5.7 leopard

python 2.5

Please provide any additional information below.

Doc\includes\mp_webserver.py leads to:

[...]
  File "[....]\multiprocessing\reduction.py", line 167, in rebuild_socket
    _sock = fromfd(fd, family, type_, proto)
  File "[....]\multiprocessing\reduction.py", line 156, in fromfd
    s = socket.fromfd(fd, family, type_, proto)
AttributeError: 'module' object has no attribute 'fromfd'

The version of processing.py 0.52 from berlios DID work on win32. Problem
is: on win32 there is no socket.fromfd; and reduction.py is supposed to
provide a substitution (which it does NOT within multiprocessing, but
within processing)

Harald

Do we need a Debian package for this ? 
I could help with that if so.

I'm using Python 2.5.4 with Leopard (10.5).  I checked out multiprocessing
from the SVN repository (Rev 73).

I'd like to know if there is a chance that the following issue gets fixed
in the backport.  I'm having a similar issue (except of course that my
target function does different things, imported modules are different,
etc.) as the post below describes:

http://developer.berlios.de/bugs/?func=detailbug&bug_id=15295&group_id=9001

Since the above report doesn't appear to get noticed, I reproduce it here:

-- BEGIN FORWARDED MESSAGE --

This issue was very hard to nail down deterministically, and it's not
entirely clear where the bug lies. Mac OSX has new restrictions on what can
be done within a fork as described in the section on "forking" in
http://developer.apple.com/releasenotes/CoreFoundation/CoreFoundation.html
. The error they discuss there looks like this:

The process has forked and you cannot use this CoreFoundation functionality
safely. You MUST exec().
Break on
__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___
YOU_MUST_EXEC__()
to debug.

To reproduce this error with processing, I cannot explain why it is these
two particular libraries but they both must be present for it to occur:
urllib2 and uuid. Here is a sample:

import multiprocessing
import urllib2
import uuid

def handle():
    urllib2.urlopen("http://www.google.com")

p = multiprocessing.Process(target=handle)
p.start()

If you remove the "import uuid", the program executes successfully, and
also executes successfully if you replace the usage of "urllib2" with
something else.

I would also note that this bug does *not* occur when using the
"multiprocessing" package in python 2.6 or 3.0.

---- END FORWARDED MESSAGE ----

I could also attach a problem report that gets spewed out (the program
actually crash) if that helps to diagnose the problem.

Taro

First, many thanks to all involved for back porting multiprocessing to
Python 2.5!

Second, in my application I need to determine if a Queue instance
is closed. I am currently using ._closed attribute for that.
Since the _closed attribute is not public then this usage is not
a good practice. So, I wonder if a method, say is_alive, could be 
introduced to determine if a Queue is instance is closed or not.

hi there,

for ease of distribution, I have create a tar-ball for the 0.0.1 release.
It is there:
 http://binet.web.cern.ch/binet/python-multiprocessing-0.0.1.tar.gz
 md5sums: db68b147b57de348cbf7cd7b37702331

(I am not super fluent with PyPI so I didn't uploaded yet)

cheers,
sebastien.

[deleted issue]

First step should probably be to backport the test_multi-processing.py
unit tests to the 0.52 API so we can figure out which core Python bug
fixes would be required to run this module on (at least) 2.4 and 2.5.

The distributing.py example (for distributing processes across machines)
given at the end of the documentation seems broken, or at least bit-rotten.
I can't even get a simple example (attached) to work: I get various weird
exceptions.

I was trying to cross-compile for powerpc and I came across an issue with
cross-compilation.

Now, a few things...

- The Makefile has `python' hardcoded into it instead of using $(RUNPYTHON).
- RUNPYTHON != PYTHON in all cases, so the assumption at the top of the
file is most likely wrong :(.
- The Makefile isn't passing in -I, -l, and -L for include directories and
lib dirs and libraries.

This issue may exist upstream too.

I'll submit a version of the Makefile which does properly cross-compile,
regardless of the consequences with broken sem_open functionality.

It's possible that all core Python bug fixes required to run multiprocessing
on older versions will get added to 2.5.3.  We would still need a patch
set for 2.5 <= 2.5.2 and for all 2.4 versions.  Those might well be 
distributed as alternate downloads from this project or picked up by
core Python for source-only releases.

Hi, I'm using a vendor supplied C++ library that spawns internal threads
and communicates amongst those using signals. Unfortunately, this wreaks
havoc upon lots of Python packages, including multiprocessing. The basic
problem is that sometimes those signals hit other threads, interrupting
system calls and causing them to fail with EINTR. If the code wasn't
written with this in mind, it is this interruption, and treats it as a
normal error. Fixes 2 places in multiprocessing, however, now allows me to
run with the supplied library in heavy use for days at a time. Ideally, I
think all system calls in multiprocessing should be made safe from this
issue, but these two instances fixed things for me.

This was tested on Ubuntu 8.04 using its stock Python2.5 and was originally
based on the svn hosted at google code r54.

As this is obviously an issue in multiprocessing in core Python, should I
take it up there, too?

This commit moved processName support from multiprocessing to the logging
module:
http://svn.python.org/view?view=rev&revision=68737

I've made a patch to test for processName in log records, and another to
re-add _check_logger_class, in multiprocessing.patch. _check_logger_class
isn't perfect, and issue 4301 will still be there, but the patches prevent
a regression.

(Issue 4301 means that logging initialization, like logging.basicConfig,
must be done after calling multiprocessing.get_logger)

hi there,

FWIW here is the link towards the cvs repository where I have been kind of
maintaining my own port of multiprocessing to py2.5:

http://atlas-sw.cern.ch/cgi-bin/viewcvs-atlas.cgi/offline/External/AtlasPyMultip
rocessing/

I shall produce a proper diff against python-multiprocessing r2, tomorrow.

hth,
sebastien.

Product: multiprocessing 2.6.2.1
Platform: Solaris 8 / Sparc
Python interpreter: python2.4
Compiler: gcc 2.95.2 (vintage, baby!)

What steps will reproduce the problem?

1. python2.4 setup.py build
2. $ ( cd build/lib.solaris-2.8-sun4u-2.4; /apps/pydev/hjoukl/bin/python2.4 -c 
'import multiprocessing' )
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "multiprocessing/__init__.py", line 87, in ?
    import _multiprocessing
ImportError: ld.so.1: /apps/pydev/hjoukl/bin/python2.4: fatal: relocation 
error: file multiprocessing/_multiprocessing.so: symbol CMSG_LEN: referenced 
symbol not found

3. Seems like Solaris 8 doesn't define these symbols. Using the recipe found 
here
http://www.mail-archive.com/[email protected]/msg01047.html
to patch multiprocessing.h:

$ /apps/pydev/bin/hg diff
diff -r 0f2ab22c260e Modules/_multiprocessing/multiprocessing.h
--- a/Modules/_multiprocessing/multiprocessing.h        Thu Sep 02 12:25:35 
2010 +0200
+++ b/Modules/_multiprocessing/multiprocessing.h        Thu Sep 02 12:30:56 
2010 +0200
@@ -52,6 +52,28 @@
 #  endif
 #endif

+/*
+ * Support for Solaris 8 (what about other old Solarises?)
+ */
+#if defined (__SVR4) && defined (__sun)
+#  include <sys/uio.h>
+#  include <stddef.h>
+#  ifndef OSSH_ALIGNBYTES
+#    define OSSH_ALIGNBYTES (sizeof(int) - 1)
+#  endif
+#  ifndef __CMSG_ALIGN
+#    define __CMSG_ALIGN(p) (((u_int)(p) + OSSH_ALIGNBYTES) &~ OSSH_ALIGNBYTES)
+#  endif
+/* Length of the contents of a control message of length len */
+#  ifndef CMSG_LEN
+#    define CMSG_LEN(len)   (__CMSG_ALIGN(sizeof(struct cmsghdr)) + (len))
+#  endif
+/* Length of the space taken up by a padded control message of length len */
+#  ifndef CMSG_SPACE
+#    define CMSG_SPACE(len) (__CMSG_ALIGN(sizeof(struct cmsghdr)) + 
__CMSG_ALIGN(len))
+#  endif
+#endif
+
 /*
  * Make sure Py_ssize_t available
  */

4. Still doesn't work after a rebuild:
$ ( cd build/lib.solaris-2.8-sun4u-2.4; /apps/pydev/hjoukl/bin/python2.4 -c 
'import multiprocessing' )
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "multiprocessing/__init__.py", line 87, in ?
    import _multiprocessing
ImportError: ld.so.1: /apps/pydev/hjoukl/bin/python2.4: fatal: relocation 
error: file multiprocessing/_multiprocessing.so: symbol sem_timedwait: 
referenced symbol not found

5. Some research indicates that sem_timedwait isn't available on Solaris 8, 
therefore patching setup.py:
$ /apps/pydev/bin/hg diff setup.py
diff -r 0f2ab22c260e setup.py
--- a/setup.py  Thu Sep 02 12:25:35 2010 +0200
+++ b/setup.py  Thu Sep 02 12:34:33 2010 +0200
@@ -65,6 +65,13 @@
         HAVE_FD_TRANSFER=1,
         )
     libraries = []
+elif sys.platform.startswith('sunos5') and os.uname()[2] == '5.8':
+    macros = dict(                  # Solaris 8 (what about older solarises?)
+        HAVE_SEM_OPEN=1,
+        HAVE_SEM_TIMEDWAIT=0,       # Not implemented
+        HAVE_FD_TRANSFER=1,
+        )
+    libraries = ['rt']
 else:                                   # Linux and other unices
     macros = dict(
         HAVE_SEM_OPEN=1,

6. Rebuild - works!

$ ( cd build/lib.solaris-2.8-sun4u-2.4; python2.4 -c 'import multiprocessing' 
&& PYTHONPATH=`pwd` python2.4 multiprocessing/tests.py )                        

test_array (__main__.WithProcessesTestArray) ... ERROR 
test_getobj_getlock_obj (__main__.WithProcessesTestArray) ... ERROR
test_rawarray (__main__.WithProcessesTestArray) ... ERROR          
test_notify (__main__.WithProcessesTestCondition) ... ok           
test_notify_all (__main__.WithProcessesTestCondition) ... ok       
test_timeout (__main__.WithProcessesTestCondition) ... ok          
test_connection (__main__.WithProcessesTestConnection) ... ok      
test_duplex_false (__main__.WithProcessesTestConnection) ... ok    
test_sendbytes (__main__.WithProcessesTestConnection) ... ok       
test_spawn_close (__main__.WithProcessesTestConnection) ... ok     
test_event (__main__.WithProcessesTestEvent) ... ok                
test_finalize (__main__.WithProcessesTestFinalize) ... ok          
test_heap (__main__.WithProcessesTestHeap) ... ok                  
test_import (__main__.WithProcessesTestImportStar) ... ERROR       
test_listener_client (__main__.WithProcessesTestListenerClient) ... ok
test_lock (__main__.WithProcessesTestLock) ... ok                     
test_rlock (__main__.WithProcessesTestLock) ... ok                    
test_enable_logging (__main__.WithProcessesTestLogging) ... ok        
test_level (__main__.WithProcessesTestLogging) ... ok                 
test_apply (__main__.WithProcessesTestPool) ... ok                    
test_async (__main__.WithProcessesTestPool) ... ok                    
test_async_timeout (__main__.WithProcessesTestPool) ... ok            
test_imap (__main__.WithProcessesTestPool) ... ok                     
test_imap_unordered (__main__.WithProcessesTestPool) ... ok           
test_make_pool (__main__.WithProcessesTestPool) ... ok                
test_map (__main__.WithProcessesTestPool) ... ok                      
test_terminate (__main__.WithProcessesTestPool) ... ok                
test_active_children (__main__.WithProcessesTestProcess) ... ok       
test_cpu_count (__main__.WithProcessesTestProcess) ... ok             
test_current (__main__.WithProcessesTestProcess) ... ok               
test_process (__main__.WithProcessesTestProcess) ... ok               
test_recursion (__main__.WithProcessesTestProcess) ... ok             
test_terminate (__main__.WithProcessesTestProcess) ... ok             
test_fork (__main__.WithProcessesTestQueue) ... ok                    
test_get (__main__.WithProcessesTestQueue) ... ok                     
test_put (__main__.WithProcessesTestQueue) ... ok                     
test_qsize (__main__.WithProcessesTestQueue) ... ok                   
test_task_done (__main__.WithProcessesTestQueue) ... ok               
test_bounded_semaphore (__main__.WithProcessesTestSemaphore) ... ok   
test_semaphore (__main__.WithProcessesTestSemaphore) ... ok           
test_timeout (__main__.WithProcessesTestSemaphore) ... ok             
test_copy (__main__.WithProcessesTestSharedCTypes) ... ok             
test_sharedctypes (__main__.WithProcessesTestSharedCTypes) ... ok     
test_synchronize (__main__.WithProcessesTestSharedCTypes) ... ok      
test_subclassing (__main__.WithProcessesTestSubclassingProcess) ... ok
test_getobj_getlock (__main__.WithProcessesTestValue) ... ERROR       
test_rawvalue (__main__.WithProcessesTestValue) ... ERROR             
test_value (__main__.WithProcessesTestValue) ... ERROR                
test_array (__main__.WithThreadsTestArray) ... ok                     
test_getobj_getlock_obj (__main__.WithThreadsTestArray) ... ok        
test_rawarray (__main__.WithThreadsTestArray) ... ok                  
test_notify (__main__.WithThreadsTestCondition) ... ok                
test_notify_all (__main__.WithThreadsTestCondition) ... ok            
test_timeout (__main__.WithThreadsTestCondition) ... ok               
test_connection (__main__.WithThreadsTestConnection) ... ok           
test_duplex_false (__main__.WithThreadsTestConnection) ... ok         
test_sendbytes (__main__.WithThreadsTestConnection) ... ok            
test_spawn_close (__main__.WithThreadsTestConnection) ... ok          
test_event (__main__.WithThreadsTestEvent) ... ok                     
test_listener_client (__main__.WithThreadsTestListenerClient) ... ok  
test_lock (__main__.WithThreadsTestLock) ... ok                       
test_rlock (__main__.WithThreadsTestLock) ... ok                      
test_apply (__main__.WithThreadsTestPool) ... ok                      
test_async (__main__.WithThreadsTestPool) ... ok                      
test_async_timeout (__main__.WithThreadsTestPool) ... ok              
test_imap (__main__.WithThreadsTestPool) ... ok                       
test_imap_unordered (__main__.WithThreadsTestPool) ... ok             
test_make_pool (__main__.WithThreadsTestPool) ... ok                  
test_map (__main__.WithThreadsTestPool) ... ok                        
test_terminate (__main__.WithThreadsTestPool) ... ok                  
test_active_children (__main__.WithThreadsTestProcess) ... ok         
test_cpu_count (__main__.WithThreadsTestProcess) ... ok               
test_current (__main__.WithThreadsTestProcess) ... ok                 
test_process (__main__.WithThreadsTestProcess) ... ok                 
test_recursion (__main__.WithThreadsTestProcess) ... ok               
test_terminate (__main__.WithThreadsTestProcess) ... ok               
test_fork (__main__.WithThreadsTestQueue) ... ok                      
test_get (__main__.WithThreadsTestQueue) ... ok                       
test_put (__main__.WithThreadsTestQueue) ... ok                       
test_qsize (__main__.WithThreadsTestQueue) ... ok                     
test_task_done (__main__.WithThreadsTestQueue) ... ok                 
test_bounded_semaphore (__main__.WithThreadsTestSemaphore) ... ok     
test_semaphore (__main__.WithThreadsTestSemaphore) ... ok             
test_timeout (__main__.WithThreadsTestSemaphore) ... ok               
test_getobj_getlock (__main__.WithThreadsTestValue) ... ok            
test_rawvalue (__main__.WithThreadsTestValue) ... ok                  
test_value (__main__.WithThreadsTestValue) ... ok                     
test_array (__main__.WithManagerTestArray) ... ok                     
test_getobj_getlock_obj (__main__.WithManagerTestArray) ... ok        
test_rawarray (__main__.WithManagerTestArray) ... ok                  
test_notify (__main__.WithManagerTestCondition) ... ok                
test_notify_all (__main__.WithManagerTestCondition) ... ok            
test_timeout (__main__.WithManagerTestCondition) ... ok               
test_dict (__main__.WithManagerTestContainers) ... ok                 
test_list (__main__.WithManagerTestContainers) ... ok                 
test_namespace (__main__.WithManagerTestContainers) ... ok            
test_event (__main__.WithManagerTestEvent) ... ok                     
test_lock (__main__.WithManagerTestLock) ... ok                       
test_rlock (__main__.WithManagerTestLock) ... ok                      
test_mymanager (__main__.WithManagerTestMyManager) ... ok             
test_apply (__main__.WithManagerTestPool) ... ok                      
test_async (__main__.WithManagerTestPool) ... ok                      
test_async_timeout (__main__.WithManagerTestPool) ... ok              
test_imap (__main__.WithManagerTestPool) ... ok                       
test_imap_unordered (__main__.WithManagerTestPool) ... ok             
test_make_pool (__main__.WithManagerTestPool) ... ok                  
test_map (__main__.WithManagerTestPool) ... ok                        
test_terminate (__main__.WithManagerTestPool) ... ok                  
test_fork (__main__.WithManagerTestQueue) ... ok                      
test_get (__main__.WithManagerTestQueue) ... ok                       
test_put (__main__.WithManagerTestQueue) ... ok                       
test_qsize (__main__.WithManagerTestQueue) ... ok                     
test_task_done (__main__.WithManagerTestQueue) ... ok                 
test_remote (__main__.WithManagerTestRemoteManager) ... ok            
test_bounded_semaphore (__main__.WithManagerTestSemaphore) ... ok     
test_semaphore (__main__.WithManagerTestSemaphore) ... ok             
test_timeout (__main__.WithManagerTestSemaphore) ... ok               
test_getobj_getlock (__main__.WithManagerTestValue) ... ok            
test_rawvalue (__main__.WithManagerTestValue) ... ok                  
test_value (__main__.WithManagerTestValue) ... ok                     
test_number_of_objects (__main__.WithManagerTestZZZNumberOfObjects) ... ok
test_answer_challenge_auth_failure (__main__.OtherTest) ... ok            
test_deliver_challenge_auth_failure (__main__.OtherTest) ... ok           
test_invalid_handles (__main__.TestInvalidHandle) ... ok                  

======================================================================
ERROR: test_array (__main__.WithProcessesTestArray)                   
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "multiprocessing/tests.py", line 862, in test_array            
    arr = self.Array('i', seq)                                        
  File "multiprocessing/__init__.py", line 260, in Array              
    from multiprocessing.sharedctypes import Array                    
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?                                      
    import ctypes                                                                      
ImportError: No module named ctypes                                             


======================================================================
ERROR: test_getobj_getlock_obj (__main__.WithProcessesTestArray)      
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "multiprocessing/tests.py", line 887, in test_getobj_getlock_obj
    arr1 = self.Array('i', range(10))                                  
  File "multiprocessing/__init__.py", line 260, in Array               
    from multiprocessing.sharedctypes import Array                     
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?                                      
    import ctypes                                                                      
ImportError: No module named ctypes                                             


======================================================================
ERROR: test_rawarray (__main__.WithProcessesTestArray)                
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "multiprocessing/tests.py", line 881, in test_rawarray         
    self.test_array(raw=True)                                         
  File "multiprocessing/tests.py", line 860, in test_array            
    arr = self.RawArray('i', seq)                                     
  File "multiprocessing/__init__.py", line 246, in RawArray           
    from multiprocessing.sharedctypes import RawArray                 
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?                                      
    import ctypes                                                                      
ImportError: No module named ctypes                                             


======================================================================
ERROR: test_import (__main__.WithProcessesTestImportStar)             
----------------------------------------------------------------------
Traceback (most recent call last):                                    
  File "multiprocessing/tests.py", line 1641, in test_import          
    __import__(name)                                                  
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?                                      
    import ctypes                                                                      
ImportError: No module named ctypes                                             


======================================================================
ERROR: test_getobj_getlock (__main__.WithProcessesTestValue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "multiprocessing/tests.py", line 823, in test_getobj_getlock
    val1 = self.Value('i', 5)
  File "multiprocessing/__init__.py", line 253, in Value
    from multiprocessing.sharedctypes import Value
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?
    import ctypes
ImportError: No module named ctypes

======================================================================
ERROR: test_rawvalue (__main__.WithProcessesTestValue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "multiprocessing/tests.py", line 817, in test_rawvalue
    self.test_value(raw=True)
  File "multiprocessing/tests.py", line 800, in test_value
    values = [self.RawValue(code, value)
  File "multiprocessing/__init__.py", line 239, in RawValue
    from multiprocessing.sharedctypes import RawValue
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?
    import ctypes
ImportError: No module named ctypes

======================================================================
ERROR: test_value (__main__.WithProcessesTestValue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "multiprocessing/tests.py", line 803, in test_value
    values = [self.Value(code, value)
  File "multiprocessing/__init__.py", line 253, in Value
    from multiprocessing.sharedctypes import Value
  File "/ae/data/pydev/DOWNLOADS/multiprocessing-2.6.2.1/build/lib.solaris-2.8-sun4u-2.4/multiprocessing/sharedctypes.py", line 10, in ?
    import ctypes
ImportError: No module named ctypes

----------------------------------------------------------------------
Ran 124 tests in 13.675s

FAILED (errors=7)

As far as I can see all the errors seem to stem from my Python 2.4 installation 
naturally not having ctypes, which was only added in Python 2.5.

I haven't tried building a more recent Python that includes the multiprocessing 
package in the stdlib but I suspect this will show similar problems for Solaris 
8 / Sparc.

This issue seems to go beyond the fix for Solaris 8 multiprocessing problems 
presented here: http://bugs.python.org/issue3110

Disclaimer: I haven't actually *used* the runnable multiprocessing build for 
anything real, apart from running the included test suite.

Patchfile attached.

Best regards,
Holger

multiprocessing-2.6.0-0.1 contains a dash in its version. This is forbidden
in rpm .spec files. Would it be possible to do something about this?

The subprocess module in Python 2.5 can redirect the output of the executed
program to an internal buffer. Example:

import os,subprocess,multiprocessing

def f():
        p = subprocess.Popen(['echo', '-n', 'pss'], stdout=subprocess.PIPE)
        out = p.communicate()
        print(repr(out))
pr = multiprocessing.Process(target=f)
pr.start()


With python2.5, this outputs (None, None). python2.6 with native
multiprocessing yields the correct ('pss', None).


Workaround: Use one of the os.popen functions:

def f():
        streams = os.popen3('echo -n psss')
        out = (streams[1].read(), streams[2].read())
        print(repr(out))
pr = multiprocessing.Process(target=f)
pr.start()

yields ('psss, '') in python2.5 and python2.6.

When I run the following simple code:
from multiprocessing import Pool

f = lambda x: x*x
p=Pool()
res = p.imap(f,range(10))

I am getting the following error both in the backport to python 2.5 and in
python 3.0:

Exception in thread Thread-1:                         
Traceback (most recent call last):                                        

  File "/usr/lib/python3.0/threading.py", line 507, in _bootstrap_inner   

    self.run()                                                            

  File "/usr/lib/python3.0/threading.py", line 462, in run                

    self._target(*self._args, **self._kwargs)                             

  File "/usr/lib/python3.0/multiprocessing/pool.py", line 225, in
_handle_tasks                  
    put(task)                                                             

  File "/usr/lib/python3.0/pickle.py", line 1317, in dumps                

    Pickler(f, protocol).dump(obj)                                        

_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup
builtins.function failed

I am using ubuntu intrepid on a centrino core2 duo based system

Hi,

could you please make a release and add the tarball to the downloads
section? I'll then create a Debian package and get it to Debian/Ubuntu, so
that it becomes the de facto standard.

Currently there is a python-processing package, but the problem is that it
has a different interface. So if people want to use multiprocessing, but
are still stuck with python2.4 or 2.5, the only option is this backport --
so let's get it into distributions.

Thanks,
Ondrej