stewori / jyni

Enables Jython to load native CPython extensions.

JyNI – Jython Native Interface

If not yet done, make sure to visit our project homepage at www.jyni.org.

Table of contents

---

1. What is JyNI?
2. Current state
3. Installing JyNI
4. Building and testing
5. Roadmap
6. Binary compatibility
7. Summary of changes to Python code
8. Copyright notice
9. License
10. Contact

---

1. What is JyNI?

A current issue of Jython hindering its wider adoption is that it does not support native extensions written for CPython like NumPy and SciPy. Since most scientific Python code fundamentally depends on exactly these native extensions, it usually cannot be run with Jython. JyNI aims to bridge this gap. It is a layer allowing Jython users to load native CPython extensions and access them from Jython the same way they would in CPython. In order to enable JyNI, you only have to put it on the Java classpath when Jython is launched. It neither requires recompiling the extension code, nor building a customized Jython fork. That means it is binary-compatible with existing extension builds.

As of this writing, JyNI does not fully implement the Python C API but it does support some important extensions, most notably the core functionalities of ctypes and NumPy.

We are constantly working to improve this support and our goal is to provide the C API needed to use the scientific stack and other extensions as soon as possible.

JyNI runs on Linux, OS-X and theoretically on various other POSIX systems (not tested). From JyNI-alpha.5 onwards, it also runs on Windows.

2. Current state

We are currently able to load a C extension into Jython, call methods and access attributes, provided that the extension uses only CPython API we have already implemented.

As of JyNI-alpha.3 JyNI performs a method for garbage collection that bridges Java's and Jython's mark-and-sweep-based approach and CPython's reference-counting-based approach as used in C extensions. JyNI's approach is explained in detail in the publication Garbage Collection in JyNI - How to bridge Mark/Sweep and Reference Counting GC., Proceedings of the 8th European Conference on Python in Science (EuroSciPy 2015), CoRR abs/1607.00825 (2016), arxiv.org/abs/1607.00825

As of JyNI-alpha.4 JyNI supports new-style classes. Support for defining new types via C API that extend existing built-in types is still limited.

As of JyNI-alpha.5 JyNI supports Windows and properly manages built-in extensions. Most notably, datetime and _winreg are now built-in. When running on Windows, JyNI enables support for the mbcs encoding in Jython.

Parse and build functions PyArg_ParseTuple(), PyArg_ParseTupleAndKeywords(), PyArg_Parse(), Py_BuildValue and related work with format strings corresponding to built-in types that are already supported. Alternatively, supported built-in types can be accessed via their C API functions.

The following built-in types are already supported:

• Number types PyInt, PyLong, PyFloat, PyComplex
• Sequence types PyTuple, PyList, PySlice, PyString, PyUnicode
• Data structure types PyDict, PySet, PyFrozenSet, PyDictProxy
• Operational types PyModule, PyFunction, PyCode, PyFrame, PyFile (partly), PyCell (partly)
• OOP types PyClass, PyInstance, PyMethod, PyClassMethod, PyStaticMethod, PyProperty
• Singleton types PyNone, PyNotImplemented, PyEllipsis, PyBool
• Native types PyCFunction, PyCapsule, PyCObject
• Iterator types PySeqIter
• Natively defined custom types
• Exception types and instances
• PyType as static type or heap type
• Weak reference types _PyWeakref_RefType, _PyWeakref_ProxyType, _PyWeakref_CallableProxyType
• New-style classes/instances

Planned:

• Support for remaining iterator types, e.g. TupleIter, ListIter
• Buffer protocol
• Support for PyByteArray (along with buffer protocol)
• Improve support for PyFile

JyNI has been tested on

• Linux Mint Debian Edition (LMDE) (32 bit)
• Linux Mint 18 (64 bit)
• Mac OS-X (10.11)
• Windows 10 (64 bit)
• Windows 7 (64 bit)

It would presumably work also on other POSIX systems. If you try it on further distributions, please consider reporting your results (see contact section).

3. Installing JyNI

This section explains how to install JyNI using prebuilt binary files. If you want to build JyNI yourself, skip to section 4. Building and testing.

• Download JyNI.jar and the binaries for your system.

• Make the binary files available to your JVM.

  • On Linux, OS-X and other systems using libLD to load binaries, place libJyNI.so and libJyNI-loader.so (OS-X: libJyNI.dylib and libJyNI-loader.dylib) somewhere on your LD_LIBRARY_PATH or tell the JVM via -Djava.library.path where to find them or place them on the Java classpath.
  • On Windows, JyNI.dll will be a folder containing python27.dll. This is due to technical reasons explained on the JyNI GSoC-2017 blog. Treat the folder JyNI.dll as if it were the binary dll file itself. It (i.e. its parent directory) must be available on the Java classpath.

• Have JyNI.jar on the Java classpath when calling Jython. For details and examples see the following section Running JyNI.

Running JyNI

To run Jython with JyNI support, call (from JyNI base dir)

java -cp jython.jar:build/JyNI.jar org.python.util.jython someFile.py

On Windows use ; instead of : and \ instead of /:

java -cp jython.jar;build\JyNI.jar org.python.util.jython someFile.py

Alternatively, one can use Jython's start script (assuming you copied it to the JyNI base directory):

jython -J-cp build/JyNI.jar

To run it from a different place, you just need to adjust the paths for JyNI and Jython in the above commands.

Note that

java -cp build/JyNI.jar -jar jython.jar

is NOT suitable to run Jython with JyNI support. It does not pass the provided classpath to Jython.

To run tests and demos, see section Test Example

4. Building and testing

• Download the sources from the latest release or clone the github repository.

• Provide Java:

  Make sure that Java 7 JDK or newer is installed.

  As of 2.7-alpha.4 JyNI attempts to guess the JDK location from the JRE location. This requires the java and javac commands to be configured properly on the path variable and assumes that java links to the JRE bundled with the JDK.

  Alternatively, you can explicitly configure the JAVA_HOME variable to point to the JDK.

  On POSIX systems, JyNI also looks for the symlink /usr/lib/jvm/default-java, which is provided by some systems (e.g. Linux MINT) and points to the specific Java folder name composed of Java version and architecture.

  If the above attempts fail for some reason, you can still adjust the JyNI makefile directly: Open the makefile appropriate for your system (located in the JyNI base folder) and adjust the variable JAVA_HOME as desired.

• Provide Jython:

  You have two ways to provide Jython. Either you copy (on non-Windows you can also (sym)link) jython.jar into the JyNI base directory or you edit makefile and adjust the JYTHON-variable to point to jython.jar.

  To use JyNIDemo.sh or other other demonstration shell scripts/batch files you need to adjust Jython path in that file(s) respectively.

  Note that current JyNI requires Jython 2.7.1.

• If not yet done, install the dev-package of Python 2.7 if you build on Linux, OS-X or likewise or install CPython 2.7 if you build on Windows. JyNI only needs pyconfig.h from that package/installation.

  Alternatively, (if you know what you're doing) you can provide your own pyconfig.h. In that case you'll have to adjust the corresponding include-path in the makefile.

• Make sure that a C compiler is installed.

  On Linux, JyNI supports building with GCC or CLANG, on OS-X only CLANG is supported.

  To build JyNI on Windows, install MSVC for Python 2.7.

• You need to have a proper implementation of the make command on your system path.

  On Windows, we use GNU Make, but CMake might work as well (not tested).

• Build JyNI:

  Open a terminal and enter the JyNI base directory.

  • for Linux/GCC type make
  • for Linux/CLANG type make -f makefile.clang
  • for OS-X type make -f makefile.osx
  • for Windows (64 bit) type make -f makefile.win64
  • for Windows (32 bit) type make -f makefile.win32

  Optionally run make clean or e.g. make -f makefile.osx clean respectively. The resulting binaries are placed in the subfolder named build.

• Make the binary files available to your JVM.

  • On Linux, OS-X and other systems that use libLD to load binaries, place libJyNI.so and libJyNI-loader.so (OS-X: libJyNI.dylib and libJyNI-loader.dylib) somewhere on your LD_LIBRARY_PATH or tell the JVM via -Djava.library.path where to find them or place them on the Java classpath.
  • On Windows, JyNI.dll will be a folder containing python27.dll. This has technical reasons explained on the JyNI GSoC-2017 blog. Treat the folder JyNI.dll as if it were the binary dll file. It (i.e. its parent directory) must be available on the Java classpath.

• For running JyNI see section Running JyNI. To run tests read the following section Test Example.

Test Example

DemoExtension is a CPython extension that demonstrates some basic features of JyNI. It can be built like an ordinary CPython extension as described in the Python tutorial.

Enter the folder DemoExtension and type python setup.py build. On Windows make sure that MSVC for Python 2.7 is installed. Optionally type python setup.py clean.

JyNI-Demo/src/test_JyNI.py runs tests for some basic JyNI features. You can run it via

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/test_JyNI.py

or ./JyNI_unittest.sh / JyNI_unittest.bat assuming that jython.jar is placed or (sym)linked in JyNI's directory. Otherwise you need to fix the path of jython.jar in the command.

JyNI-Demo/src/JyNIDemo.py demonstrates the use of DemoExtension from Python side. It should run perfectly with either CPython 2.7.x or Jython+JyNI. To run it, type

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIDemo.py

You can alternatively run ./JyNIDemo.sh / JyNIDemo.bat.

To see a basic demonstration that JyNI is capable of using the original datetime module (a built-in module as of JyNI-alpha.5), run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIDatetimeTest.py

To see a demonstration of exception support (on native level), run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIExceptionTest.py

To see a demonstration of Tkinter support, run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNITkinterTest.py

or ./JyNITkinterDemo.sh / JyNITkinterDemo.bat.

For a demonstration of ctypes support, run

java -cp jython.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIctypesTest.py

or ./JyNIctypesDemo.sh / JyNIctypesDemo.bat.

5. Roadmap

Further steps:

• improve NumPy support
• support buffer protocol
• support CFFI and SciPy
• support further extensions
• JyNI 3

6. Binary compatibility

CPython extensions must be compiled with the flag WITH_PYMALLOC activated (which is actually the default anyway). Otherwise they must not use the macros PyObject_MALLOC, PyObject_REALLOC and PyObject_FREE directly. If they use these macros with WITH_PYMALLOC not activated, JyNI will most likely produce segmentation faults. Without the flag these macros directly map to the system's malloc function family. In that case JyNI would not be able to prepare the JyObject header in every case. However, it would be easy to recompile such extensions for JyNI. Simply add the WITH_PYMALLOC definition at compile time or replace the macros by their corresponding function calls.

In general, the less an extension hacks with CPython-specific internals, the greater the chance it will run with JyNI. Especially allocation/deallocation should not be done by hand, since JyNI must be able to setup the JyObject headers.

We hope that most extensions are not affected by this issue.

7. Summary of changes to Python code

Briefly, we took the Python files crucial for loading native C extensions and modified them to perform the explained purpose of JyNI. Mainly we changed alloc and dealloc behavior of PyObjects consequently, also when inlined like in stringobject.c or intobject.c. The new allocation behavior adds a header called JyObject before the PyObject header or even in front of PyGC_Head if present. This header contains information that allows to wrap or mirror a corresponding Jython jobject in a seamless way.

Mirror mode is used for built-in types that allow access to their data structure via macros. To keep these accesses valid, the data structure mirrors the actual Jython jobject. Syncing is not an issue for immutable objects and can be done initially for each object. One mutable built-in object needing mirror mode is PyList. We perform sync from Java by inserting a special java.util.List implementation as a backend into the corresponding Jython PyList. A bit more tricky is PyByteArray, which also features a direct access macro, namely PyByteArray_AS_STRING. However, a fix for this has low priority, as this macro is not used by NumPy. Nevertheless, we can say a bit about this issue. As PyByteArray in Jython uses a primitive byte[] array as backend, we cannot replace the backend by a subclass. Our most promising idea to solve the issue anyway is to insert a custom implementation of SequenceIndexDelegate as delegator. PyByteArray is a subclass of PySequence and thus offers a delegator field that allows to customize the indexed access behavior.

Built-ins not providing access via macros can be wrapped. The original CPython data structure is not used and not even allocated. All functions of such objects are rewritten to delegate their calls to Jython. However, also in these cases, we usually do not rewrite everything. Functions that access the data only via other functions are mostly kept unchanged.

8. Copyright notice

Copyright of Python and Jython: Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 Python Software Foundation. All rights reserved.

Copyright of JyNI: Copyright (c) 2013, 2014, 2015, 2016, 2017, 2018 Stefan Richthofer. All rights reserved.

9. License

The software in this package is distributed under the GNU Lesser General Public License.

A copy of GNU Lesser General Public License (LGPL) is included in this distribution in the files COPYING and COPYING.LESSER. If you do not have the source code, it is available at:

https://github.com/Stewori/JyNI

JyNI is partly based on source files from CPython 2.7.3, 2.7.4, 2.7.5, 2.7.6, 2.7.7, 2.7.8, 2.7.9, 2.7.10, 2.7.11, 2.7.12 and 2.7.13. As such, it includes the common license of CPython 2.7.3, 2.7.4, 2.7.5, 2.7.6, 2.7.7, 2.7.8, 2.7.9, 2.7.10, 2.7.11, 2.7.12, 2.7.13 and Jython in the file PSF-LICENSE-2. Whether a source file is directly based on CPython source code is documented at the beginning of each file.

For compliance with PSF LICENSE AGREEMENT FOR PYTHON 2, section 3, the required overview of changes done to CPython code is given in section 7 of this file.

For convenience, a copy of the current section is provided in the file LICENSE.

10. Contact

Please use the contact information provided on www.jyni.org/#contact.

jyni's Issues

NumPy 1.14+ compatibility

... is currently broken:

Traceback (most recent call last):
  File "/data/workspace/linux/JyNI/JyNI-Demo/src/JyNINumPyTest.py", line 68, in <module>
    import numpy as np
  File "/data/workspace/linux/numpy/1.14.0/numpy/__init__.py", line 168, in <module>
    from . import ma
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/__init__.py", line 44, in <module>
    from . import core
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 6333, in <module>
    masked = masked_singleton = MaskedConstant()
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 6270, in __new__
    cls.__singleton = MaskedArray(data, mask=mask).view(cls)
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 2790, in __new__
    _data = ndarray.view(_data, cls)
AttributeError: 'type$1' object has no attribute 'update'

Stick to NumPy 13.3 until this is solved...

Running multiarray with JyNI

Hi,

I am trying to import multiarray as a next step towards numpy support. As test I use the following snippet:

import sys

sys.path.append('/usr/lib64/python2.7/lib-dynload')
sys.path.append('/usr/lib64/python2.7/site-packages/numpy')
sys.path.append('/usr/lib64/python2.7/site-packages/numpy/core')

print 'before multiarray import'

import datetime

import multiarray

print 'after multiarray import'

a = multiarray.array([1, 2, 3])

print a

Running this script usually fails with something like the following besides some rare cases where it actually works: (I can also provide the hs_err files if needed.)

> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00007f4998fe073c, pid=11479, tid=139954600363776
> #
> # JRE version: OpenJDK Runtime Environment (8.0_72-b15) (build 1.8.0_72-b15)
> # Java VM: OpenJDK 64-Bit Server VM (25.72-b15 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C  [libJyNI.so+0xe073c]  pushExStack+0xa9
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/<user>/programming/jyni/JyNI/hs_err_pid11479.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> 

This looks like some side effect due to some memory, which is overwritten. It looks like this is caused in the _PyImport_LoadDynamicModuleJy, when the "dyn load func" is called. I would appreciate any hints about this before I dig deeper into this.

Thanks,

Lothar

Can't seem to load the co_stacksize extension

import pyspark
File "/usr/lib/python2.7/site-packages/pyspark/init.py", line 51, in
from pyspark.context import SparkContext
File "/usr/lib/python2.7/site-packages/pyspark/context.py", line 31, in
from pyspark import accumulators
File "/usr/lib/python2.7/site-packages/pyspark/accumulators.py", line 97, in
from pyspark.serializers import read_int, PickleSerializer
File "/usr/lib/python2.7/site-packages/pyspark/serializers.py", line 72, in
from pyspark import cloudpickle
File "/usr/lib/python2.7/site-packages/pyspark/cloudpickle.py", line 145, in
_cell_set_template_code = _make_cell_set_template_code()
File "/usr/lib/python2.7/site-packages/pyspark/cloudpickle.py", line 109, in _make_cell_set_template_code
return types.CodeType(
AttributeError: 'tablecode' object has no attribute 'co_stacksize'

Testing problems

Hi,

I have problems in the testing. When I run the datatime test, I got an error relative to the function leaveRecursiveCall. This is the output:

---

java -cp ../jython-standalone-2.7-b3.jar:build/JyNI.jar org.python.util.jython JyNI-Demo/src/JyNIDatetimeTest.py
[sudo] password for felix:
Traceback (most recent call last):
File "JyNI-Demo/src/JyNIDatetimeTest.py", line 15, in
import datetime
at JyNI.JyNI.initJyNI(Native Method)
at JyNI.JyNI.(JyNI.java:138)
at JyNI.JyNIImporter.load_module(JyNIImporter.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: leaveRecursiveCall

What am I missing?
Thanks,
Félix

Iterator support (TypeError: argument of type 'dict' is not iterable)

This is exciting and looks very promising! I was able to download the binaries and run them on my machine (Arch Linux). I was able to import numpy and tested basic functionality, e.g. this works:

import numpy
img = numpy.zeros( (3, 4) ) + 3
random = np.random.rand()

I was not able to generate a random int, though:

>>> np.random.randint(3)
Traceback (most recent call last):
  File "mtrand.pyx", line 970, in mtrand.RandomState.randint (numpy/random/mtrand/mtrand.c:15903)
TypeError: argument of type 'dict' is not iterable

This is a migration to a separate issue from #2 . Following up on @Stewori's comment I looked for PyDict_Iter but my search was not successful. My best guess would be to get the iterator from the list returned by PyDict_Items when requested. Maybe that happens already and it will be resolved automatically once ListIter is supported.

I am not very familiar with the Python C API but I would definitely have a look into it. If you could point me to where the iterator methods for the python objects are defined, that would make it a lot easier for me!

You can probably add Arch Linux to the systems JyNI has been tested on. I did not build it myself, though.

If you are interested in why we would like to use JyNI: I created a compatibility layer between numpy and imglib2 based on pyjnius: imglib2-imglyb. This library is the core for modern ImageJ and with that compatibility layer we try to get the best of both worlds. The python code of imglyb needs only slight modificatoins to work with JyNI, and shipping JyNI with Fiji would probably be much easier than shipping a whole CPython distribution, so we are very interested in what happens with JyNI.

Numpy Support

Hey,

can you lay out some steps to get my feet wet in direction of Numpy Support? Maybe some small "junior" jobs would help in general, since this both involves Python-C-binding and Jython/JVM-C-binding experience.

Cheers,
Christian

PyFile support

I have explored supporting matplotlib font loading, which require PyFile support.

I have commented in

FILE *
PyFile_AsFile(PyObject *f)
{
    printf("Accessing file.\n");
    if (f == NULL || !PyFile_Check(f))
        return NULL;
    else
        return ((PyFileObject *)f)->f_fp;
}

Which is called and immediately SIG_ABORTS. I get a dump file from the JVM:

Register to memory mapping:

RAX=0x00007fb7cbdd8280: PyFile_Type+0 in /home/christian/Development/JyNI/build/libJyNI.so at 0x00007fb7cbaa9000
RBX=0x00007fb7f8070958 is an unknown value
RCX=0x00007fb7c055d390 is an unknown value
RDX=0x0000000000000001 is an unknown value
RSP=0x00007fb8278190e8 is pointing into the stack for thread: 0x00007fb82000a000
RBP=0x00007fb7f8032e30 is an unknown value
RSI=0x00007fb7c055d390 is an unknown value
RDI=0x00007fb7f8032e30 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x0000000000000004 is an unknown value
R10=0x0000000000000319 is an unknown value
R11=0x00007fb7cbb08b65: PyErr_SetString+0 in /home/christian/Development/JyNI/build/libJyNI.so at 0x00007fb7cbaa9000
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x00007fb827819130 is pointing into the stack for thread: 0x00007fb82000a000
R15=0x0000000000000001 is an unknown value


Stack: [0x00007fb82771f000,0x00007fb827820000],  sp=0x00007fb8278190e8,  free space=1000k
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3935  JyNI.JyNI.callPyCPeer(JLorg/python/core/PyObject;Lorg/python/core/PyObject;J)Lorg/python/core/PyObject; (0 bytes) @ 0x00007fb81183612c [0x00007fb8118360c0+0x6c]
j  JyNI.PyCPeerType.__call__([Lorg/python/core/PyObject;[Ljava/lang/String;)Lorg/python/core/PyObject;+36
J 2911 C1 org.python.core.PyObject.__call__(Lorg/python/core/PyObject;)Lorg/python/core/PyObject; (16 bytes) @ 0x00007fb811ae0fec [0x00007fb811ae0e80+0x16c]
J 5454 C2 org.python.core.PyObject.__call__(Lorg/python/core/ThreadState;Lorg/python/core/PyObject;)Lorg/python/core/PyObject; (6 bytes) @ 0x00007fb8125b22e0 [0x00007fb8125b22a0+0x40]
j  matplotlib.font_manager$py.createFontList$16(Lorg/python/core/PyFrame;Lorg/python/core/ThreadState;)Lorg/python/core/PyObject;+673
j  matplotlib.font_manager$py.call_function(ILorg/python/core/PyFrame;Lorg/python/core/ThreadState;)Lorg/python/core/PyObject;+368

Just returning NULL is not sufficient as a quick hack and I guess that I need to wrap a Jython class for PyFile instead of accessing the pointer directly. You said it would be easy to fix, but in the comment you say:

//Providing the following method via JyNI would be very hard or impossible.
//However, we could create a FILE-Pointer to the same file and also adjust
//io mode and seek-position to the values of the given PyFile.
//But this may lead to conflicts in systems not supporting multiple file
//access (i.e. windows) and would not stand equality-checks via pointer-wise "==".

I had some frustrations with filesystem APIs lately for my kv-store implementation, so I would also be careful with double access. What are your suggestions to proceed? Also do you have some chat channel? This might be easier to get started.

support for the BufferProtocol

PyByteArray would be the next step towards PyByteArray, BufferProtocol, SciPy and Pandas support. It should be implemented in context of overall support for the BufferProtocol

Add debug target to makefile.

In #10 it was explained how to build JyNI in debug mode. That requires editing the makefile so we should better have a build target for debug. I created this issue to keep track of that idea.

Name JyNI.jar Mandatory

As you said, it's better to create a new issue,

When u speak about Nexus u probably refer to the phone, but me i refer to Maven and his nexus
(this : http://www.objis.com/formation-java/tutoriel-maven-deploiement-archiva.html)

I can confirm this to you, when u rename the JyNI.jar like Jyni.jar or JyNITEST.jar it doesn't work
I tried many times before contacting you

JyNI need to be called JyNI.jar but i don't know why

I've a Netbeans project which can confirm this to you if u want

Request to explore JavaCPP as an extension to model Python FFI.

Hi @Stewori

This is not an issue so please do feel free to close it down. This is more like a query/suggestion regarding the project. The basic question is -

Why is a seperate JyNI project needed to accomplish the linkage to python native extensions?

I'd like to suggest that perhaps, https://github.com/bytedeco/javacpp might be a worthy option to explore in this space as well. It's being used by ND4J and DeepLearning4J projects plus there are a number of bindings already generated located here https://github.com/bytedeco/javacpp-presets .

I hope that JavaCpp might take us closer to a proper package compatibility. I'm really looking forward to using proper python packages with Jython , Thanks again :)

Jython 2.7.2, NumPy advice

The recent release of Jython 2.7.2b2 breaks JyNI support.
Luckily, I was able to fix this in Jython, so there is no new JyNI release necessary for this issue. However, the fix is not included in Jython 2.7.2b2 but will be in the final 2.7.2 release (and in further betas if there should be any).

Even with this fix in place, I noticed that NumPy 1.13.3 is broken. However, NumPy 1.13.1 seems to work like usual. (1.13.2 and 1.x.y with x > 13 are broken anyway.)

So, to keep using NumPy with JyNI and Jython 2.7.2, stick to Jython master, 2.7.2b3+ or final release and role back from NumPy 1.13.3 to 1.13.1.

Setting JyNI to work with Processing

Dear @Stewori ,

Thank you for the clarifications provided on the Processing.py git thread.

I (and some other folks from the Processing community) would love to see JyNI working in Processing Python mode but are completely overwhelmed by the installation instructions. Would you mind give us a hand with this ?

Here's what I could do so far:

• looked for the jython.jar file on my computer. It lies in a Processing sub-folder called 'mode' with 2 other jar files, PythonMode.jar and guava-17.0.jar

• set a JRE_HOME variable pointing to the JRE installation path C:\Program Files (x86)\Java\jre1.8.0_211

This leaves me with 2 questions:

• Do I need to put JyNI.jar and JyNI.dll in this 'mode' sub-folder (with the other jar files) or somewhere inside the JRE installation, like in C:\Program Files (x86)\Java\jre1.8.0_211\lib for example ?
• How do I make these 2 files available to the Java classpath ? (That's the million dollar question for us)

Your help, even guidelines or a simple hint, would be greatly appreciated.

Sincerely,

solub

JyNI-C/src/Python/pythonrun.c:1956 isnt' know struct sigaction context ocontext

JyNI-C/src/Python/pythonrun.c:1956:19: error: storage size of ‘context’ isn’t known
struct sigaction context, ocontext;
^
JyNI-C/src/Python/pythonrun.c:1956:28: error: storage size of ‘ocontext’ isn’t known
struct sigaction context, ocontext;
^
JyNI-C/src/Python/pythonrun.c:1961:10: error: ‘SIG_ERR’ undeclared (first use in this function)
return SIG_ERR;

Paths to subpackages are not resolved by find_module in JyNIImporter

Subpackagenames are not resolved by find_module in JyNIImporter. The reason is that find_module does not descent into the directories.

This is also relevant for local import paths (from . import multiarray) since they are changed to global paths (numpy.core.multiarray) and handed over to find_module as name.

I implemented a solution for this and will upload it after further testing.

Tkinter on OSX

Hi all,

I've successfully compiled JyNI on OSX and none of the unit tests is failing.
But there's trouble with the Tkinter demo: java starts an osx application (with the Tk icon), that shows no window and is not responding at all.

Has anyone experienced the same issue?
I'll look into the problem and try to collect all information about it in this thread.

Best,
Malte

Under memory stress JVM builds an invalid native jobject, potentially causing segfault during GC

This issue is sourced out from #32, see #32 (comment) and #32 (comment).
Reproducible by this code:

import sys

sys.path.insert(0, '/data/workspace/linux/numpy/1.13.3')

import numpy as np
from java.lang import System
import time

for i in range(2000):
	print i
	np.array(range(7))

System.gc()
time.sleep(2)

It may or may not be possible to reproduce this somehow without NumPy or even without JyNI. If the issue triggers it results in the following log (excerpt), pointing to JyAttribute.delAttr:

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 2143 C1 org.python.core.JyAttribute.delAttr(Lorg/python/core/PyObject;B)V (169 bytes) @ 0x00007f19297499bb [0x00007f19297496c0+0x2fb]
J 2148 C1 JyNI.JyNI.clearNativeHandle(Lorg/python/core/PyObject;)V (37 bytes) @ 0x00007f192974d02c [0x00007f192974cd60+0x2cc]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x666a0b]
V  [libjvm.so+0x6886fe]
V  [libjvm.so+0x68b02f]
C  [libJyNI.so+0x46f7e]  JyNI_CleanUp_JyObject+0xfc
C  [libJyNI.so+0xf42fe]  meth_dealloc+0x240
C  [libJyNI.so+0xfc87b]  JyGC_clearNativeReferences+0xefa
C  [libJyNI-Loader.so+0x561c]  Java_JyNI_JyNI_JyGC_1clearNativeReferences+0x34
j  JyNI.JyNI.JyGC_clearNativeReferences([JJ)Z+0
j  JyNI.gc.JyWeakReferenceGC$GCReaperThread.run()V+251
v  ~StubRoutines::call_stub
V  [libjvm.so+0x666a0b]
V  [libjvm.so+0x663fe4]
V  [libjvm.so+0x6645c7]
V  [libjvm.so+0x6a78a4]
V  [libjvm.so+0xa15547]
V  [libjvm.so+0xa15a0c]
V  [libjvm.so+0x8bae12]
C  [libpthread.so.0+0x76ba]  start_thread+0xca

This is caused by some native jobject that should be representing a org.python.core.PyObject actually does not. Still, GetObjectRefType from JNI API reports the object as a valid reference of type JNIWeakGlobalRef. The object is not NULL and not null. Calling getClass on the object on Java side results in java.lang.object. Apparently, no Java-side exception is pending. If there was one, it got cleared.

The issue can be fixed in symptomatic sense by changing JyNI.clearNativeHandle to the following:

clearNativeHandle(PyObject object) {
		if (object == null) {
			System.err.println("JyNI-Warning: clearNativeHandle called with null!");
			return;
		}
		if (object instanceof PyCPeer)
			((PyCPeer) object).objectHandle = 0;
		else {
			if (!(object instanceof PyObject))
				System.err.println("JyNI-Warning: clearNativeHandle received Non-PyObject!");
			else
				JyAttribute.delAttr(object, JyAttribute.JYNI_HANDLE_ATTR);

However we should investigate under which circumstances the invalid jobject is created. I am somewhat confident that native JyNI code never creates a plain java.lang.object. Is it possible that the JVM creates faulty native jobjects under heavy memory stress or if it runs out of native references? (c.f. EnsureLocalCapacity in JNI API) This suspect is supported by the observation that the issue can be avoided by calling System.gc right after import numpy:

import numpy as np
System.gc()
time.sleep(2)

The JVM does not take natively used memory into account to decide when it's time to run GC. Maybe we should trigger GC explicitly in JyNI from time to time. E.g. after importing a native module or if the number of JyGCHeads exceeds some limit. A proper strategy needs to be figured out.

PyExc_TypeError has ob_type == NULL

I have been investigating #13, and I found that numpy.random.randint() segfaults occasionally. I'm not sure why it occurs so rarely or what the root cause is, but (after finding myself reading assembly) the actual seg fault is connected to PyObject_Call being passed PyExc_TypeError as the function to be called. This then causes a segfault as soon as a property of the ob_type is accessed (in this case: ob_type->tp_flags). I have written a simple C function that reliably causes this bug:

static PyObject* test_TypeError(PyObject *self, PyObject *args){
	PyObject* tup = PyTuple_Pack(1, Py_BuildValue("s","Hello"));
	PyObject_Call(PyExc_TypeError, tup, NULL);
	return tup;
}

CPython happily runs this (when it is part of a C extension), while JyNI segfaults. In JyNI, for PyExc_TypeError->ob_type==NULL, I think this is the cause of the segfault. I haven't been able to work out if PyExc_TypeError in CPython has ob_type==NULL. However, I can't find any checks for ob_type==NULL throughout the python codebase, which suggests the property should never be NULL. I did find a check in JyNI for this, and it has this comment just before it:

//JyNI note: if ob_type is NULL, we assume that base is a Type object that was
//not initialized because of our use-on-demand philosophy regarding PyType_Ready.

I don't quite understand what this means but it seems related, could you (@Stewori) possibly explain a bit about what is happening/what this means?

This could be solved in this case by working around it in the PyObject_Call function, it's possible to work out you are dealing with an error/exception and figure out what type without accessing any properties of ob_type. But if ob_type should never be NULL, perhaps it would be better to solve this more generally instead of trying to work around it.

With regards to numpy.random.randint(), it is possible that this error is always caught internally since it doesn't seem to set an error message or call the type error properly. Maybe in JyNI it just segfaults before it can be caught. But I'm not entirely sure, I haven't been able to track the particular problem down as it seems to be affected by timing (so adding print statements or running in debug mode make it less likely to occur), and it is random when it occurs(so I need to run numpy.random.randint() millions of times to get the error to appear).

'getset_descriptor' object is not callable at 1668:0 in defchararray.py

The error I'm getting when running:

import numpy as np

is:

TypeError: Error when calling the metaclass bases
    'getset_descriptor' object is not callable at 1668:0 in defchararray.py

This occurs with the following setup:

• JyNI 2.7-alpha5
• Jython 2.7.1
• numpy 1.13.1
• macOS 10.12
• JDK 8u144

What can I do to help troubleshoot further? Am I missing a dependency?

Matplotlib Support

Factoring out from #11

Yes, it is reproducible. I have just tried to find the part in matplotlib where it is called to shrink the case, but it is a bit messy with all the backend monkey patching. I am not sure where show() calls into multiarray (and why), I can instrument with print debugging, but this is a bit inefficient. I tried to use the Emacs Python debugger with CPython, but I have had no success yet. Any suggestions? How do you pin things like that down?

Since the crash happens in multiarray, and matplotlib uses numpy, I think it is a numpy issue.

SciPy

Hello @Stewori

I'm currently using JyNI to import Numpy in a Java APP. but now i need to import SciPy or Scikit-Learn

I would like to know if there is a way to import them, or if JyNI need an update to import them

if it need an update could u explain me how to do it? or are u planning to do it?

How to correcty install and use JyNI with numpy support on Jython?

Hello,

I don't understand well the sequence of steps to start usage of JyNI with numpy (Windows platform). I already created an issue on Jython site:
https://bugs.jython.org/issue2930
But I'm not sure that it is a correct place, maybe it is question to you.

I read your paper https://arxiv.org/pdf/1607.00825v1.pdf and tried to reproduce the simplest example:

import sys
sys.path.append(’path_to_numpy1.12_pre-release’)
import numpy as np
a = np.array([2, 5, 7])
...

But what is here the 'path_to_numpy1.12'? Where should I place the content of your archive JyNI-2.7-alpha.5-bin-win-amd64.zip ? Now I just unpacked it and placed both files into C:\JyNI , then added this directory to sys.path.append. But it seems that it is not a right way. Results are the following:

Traceback (most recent call last):
File "C:/siams/computer-vision/stare-python/jython-experiments/net/algart/jython/tests/SimpleJythonNumpyTest.py", line 3, in
import numpy as np
File "C:\jython2.7.2\Lib\site-packages\numpy_init_.py", line 142, in
from . import core
File "C:\jython2.7.2\Lib\site-packages\numpy\core_init_.py", line 71, in
raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:

• If you're working with a numpy git repository, try git clean -xdf
  (removes all files not under version control) and rebuild numpy.
• If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
• If you have already reinstalled and that did not fix the problem, then:

  1. Check that you are using the Python you expect (you're using C:\jython2.7.2\bin\jython.exe),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.

  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues. Please include details on:

     • how you installed Python
     • how you installed numpy
     • your operating system
     • whether or not you have multiple versions of Python installed
     • if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: No module named _multiarray_umath

Nested imports cause deadlock

Nested imports cause deadlocks. The deadlock is caused by native libraries which are loading (indirectly) other native libraries before calling "LEAVE_JyNI". Then the next JyNI_loadModule (JyNI.c) call will stop at ENTER_JyNI.

One possible solution might be to add LEAVE_JyNI and ENTER_JyNI around the call of the dynamic load function (return value of _PyImport_GetDynLoadFunc) in _PyImport_LoadDynamicModuleJy (importdl.c) i.e. around line (*p)();

Another solution might be to check in loadModule in JyNI.java, if one was called from the native library and then calls LEAVE_JyNI and at the end of the function restores the state by calling ENTER_JyNI.

Stefan, I would be happy to hear your opinion on that, since I do not know what requirements on this locking mechanism exist.

Using JyNI with Jython 2.0-final

Hi, is JyNI OK to use with Jython 2.0-final? The final release came out in May, but JyNI says it needs Jython 2.0-b4.

How to setup a development environment?

I have found out when I was trying to debug numpy that building all stuff JyNI related can be hairy. Could you provide a guide (and maybe a short screencast)?

When using numpy, we call np.split(x,2),np.hsplit(x,2),np.vsplit(x,2), x is a numpy array. But the command didn't output anything or report any mistakes

However when I replaced np.hsplit(x,2) with np.hsplit(x,[2]), it can work . So I wonder how can I use np.hsplit(x,2)

umath fails to load

umath fails to load with the message: AttributeError: _ARRAY_API not found

Cause for this is a bug in JyNI-C/src/Python/import.c in the function PyObject * PyImport_ImportModule(const char *name). The current implementation calls directly the import method from Jythons' builtin class. This leads to the fact, that function calls like
PyImport_ImportModule("A.B.C")
will return the top module A instead of A.B.C as it should be.

In umath this is location in the numpy/core/src/umath/umathmodule.c file within the numpy package
. In the function initumath in the line

PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");

the variable numpy will ironically be the numpy package and not the multiarray subpackage. Which leads to the observed error, if later the attribute "_ARRAY_API" is checked with this line:

c_api2 = PyObject_GetAttrString(numpy, "_ARRAY_API");
The solution to this bug is to change the function PyImport_ImportModule(const char name) to call PyImport_Import(PyObject module_name) as in the cpython implementation and uncomment the original PyImport_Import(PyObject* module_name) in JyNI-C/src/Python/import.c and let it call the Jython methods after the silly_list has been built up.

In order to do this a comment how these lines

r = PyObject_CallFunction(import, "OOOOi", module_name, globals,
                              globals, silly_list, 0, NULL);

should be called in Jython would be very helpful.

custom module imported but functions not exported

Hi,

I wrote my custom module in C for Python and I can imported like this:
java -cp jython-standalone-2.7.2.jar:Lib/JyNI.jar org.python.util.jython test.py
The problem is that my functions are not exported and I can't call them allthough the module is successfully imported.

test.py contains the following:

import mtrand print(dir(mtrand)) print(mtrand.__file__)

and the output is
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'mtrand']
If I run the same thing under python2 I can see all the exported funcitons:

`python
Python 2.7.18 (default, Mar 8 2021, 13:02:45)
[GCC 9.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import mtrand
dir(mtrand)
['author', 'builtins', 'doc', 'file', 'name', 'package', 'test', 'mt_rand', 'mt_rand_ab', 'mt_rand_orig', 'mt_srand']

`

So, why are my functions not exprted with JyNI?

Thank you!

mmap: String access is truncated or causes NullPointerException

In the first pic, i used mmap to read file with python. the second is jython and JyNI. the string is truncated when i used JyNI.

nltk download crashes

To import nltk I need to import numpy first, otherwise I get:

>>> import nltk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/nltk/__init__.py", line 89, in <module>
    from nltk.internals import config_java
  File "/usr/lib/python2.7/dist-packages/nltk/internals.py", line 31, in <module>
    from nltk import compat
  File "/usr/lib/python2.7/dist-packages/nltk/compat.py", line 51, in <module>
    class UTC(tzinfo):
TypeError: Error when calling the metaclass bases
    'getset_descriptor' object is not callable

The tk download dialog works fine, but it crashes the JVM when I select something and hit "download":

>>> import numpy
>>> import nltk
>>> nltk.download()
showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml
*** Error in `java': free(): invalid pointer: 0x00007fe2adc24cb8 ***
fish: “java -cp jython.jar:build/JyNI.…” terminated by signal SIGABRT (Abort)

Patch: NumPy 13.2 broken for JyNI 2.7-alpha.5

This is an issue with mtrand.c, which is a Cython-generated file. In the mtrand folder, NumPy bundles a script called generate_mtrand_c.py. This is used to generate mtrand.c, in NumPy 13.2 prebuilt with Cython 0.27. If you build it using Cython 0.25 everything works fine (0.26 not tested).
For convenience I prepared a Cython-0.25-built version of mtrand.c.

You can either replace mtrand.c if you build NumPy by yourself or use the binary mtrand.so (Linux 64 only as of this writing; I will add more binaries soon). You can put the binary on sys.path such that it preceeds NumPy. Then it should override the original mtrand.so.

Sure, this needs to be fixed in JyNI, but I cannot tell when I'll find time. So, until then use this workaround for NumPy 13.2.