Fast conversion from Java byte array to Numpy uint8 array about jpype

[marscher]

As I wrote some speedup improvements for array wrapping, I thought about directly returning numpy arrays, but unfortunately did not find the time to implement it. It should be relatively easy to implement.
If you use the slice operator on the java array

 np.array(buf[:])

it should be fast in comparison to the single element access, which is performed in the non sliced access mode. This is due the overhead of iterating over every element and wrap it in two objects, one for Python and for jpype. But I just discovered a bug in the sliced access pattern for byte type. Will provide a fix soon.

[marscher]

now its fixed, but it is still 300 times slower, than direct memory access, since a python object needs to be created for every element. Creating numpy arrays on C-level should be super easy and super fast!

[martinandersen]

Thanks for the quick response! I found out that the following hack works and is quite fast as well, but of course it is not very clean:

java_str = jpype._jclass.JClass('java.lang.String')(buf, 'ISO-8859-1') 
arr = numpy.array(numpy.frombuffer(buffer(java_str.toString()),dtype='uint16'),dtype='uint8')

The possibility to create numpy arrays by means of direct memory access would be a very nice addition!

[originell]

Just for completeness sake: related commit f060025

[marscher]

this hack is pretty cool, wonder if it also works with double arrays converted to a byte array 👍

[martinandersen]

This is actually kind of what I'm doing. I am importing different types of data by (i) copying the raw data from a Java array to an uint8 numpy array, and then (ii) converting the uint8 numpy array to a numpy array of another type. For example, to "interpret" the bytes as 32-bit little-endian floats, I use the following line:

arr_float32 = numpy.frombuffer(buffer(arr),dtype='<f')

[marscher]

well if I try to directly create a buffer object from a java double array, it fails with "TypeError: buffer object expected" and converting the array to a java string doesn't work neither.
Or If one tries to wrap a double[] in a DoubleBuffer, how do one convert this back to a ByteBuffer?
Would be nice, if you can provide some more detail.

[martinandersen]

I think that I misunderstood your comment about double arrays. I am basically reading the contents of a file into a byte array buf, and then I use jpype to create a numpy uint8 array that I convert to whatever data type the original file contains. In other words, I am only using byte arrays in Java regardless of the data type contained in the file. This is what my code looks like when the byte array buf contains the bytes from a file with little-endian 32-bit floats:

java_str = jpype._jclass.JClass('java.lang.String')(buf, 'ISO-8859-1') 
arr = numpy.array(numpy.frombuffer(buffer(java_str.toString()),dtype='uint16'),dtype='uint8')
arr_float32 = numpy.frombuffer(buffer(arr),dtype='<f')

I am not sure if there is a fast way to convert a Java double array to a Java byte array. This would be a simple type cast in C, but in Java you may need to use a method like ByteBuffer.putDouble().

[marscher]

I believe your way of conversion may still be a bit faster than the code in #73, but not that hackish and so hoping you are willing to check it out 👍

[martinandersen]

What a nice improvement. On my machine it is around twice as fast as the string conversion hack, and it produces readable code!

[marscher]

will close this now, since the changes are merged