Comments (11)
As I wrote some speedup improvements for array wrapping, I thought about directly returning numpy arrays, but unfortunately did not find the time to implement it. It should be relatively easy to implement.
If you use the slice operator on the java array
np.array(buf[:])
it should be fast in comparison to the single element access, which is performed in the non sliced access mode. This is due the overhead of iterating over every element and wrap it in two objects, one for Python and for jpype. But I just discovered a bug in the sliced access pattern for byte type. Will provide a fix soon.
from jpype.
now its fixed, but it is still 300 times slower, than direct memory access, since a python object needs to be created for every element. Creating numpy arrays on C-level should be super easy and super fast!
from jpype.
Thanks for the quick response! I found out that the following hack works and is quite fast as well, but of course it is not very clean:
java_str = jpype._jclass.JClass('java.lang.String')(buf, 'ISO-8859-1')
arr = numpy.array(numpy.frombuffer(buffer(java_str.toString()),dtype='uint16'),dtype='uint8')
The possibility to create numpy arrays by means of direct memory access would be a very nice addition!
from jpype.
Just for completeness sake: related commit f060025
from jpype.
this hack is pretty cool, wonder if it also works with double arrays converted to a byte array š
from jpype.
This is actually kind of what I'm doing. I am importing different types of data by (i) copying the raw data from a Java array to an uint8 numpy array, and then (ii) converting the uint8 numpy array to a numpy array of another type. For example, to "interpret" the bytes as 32-bit little-endian floats, I use the following line:
arr_float32 = numpy.frombuffer(buffer(arr),dtype='<f')
from jpype.
well if I try to directly create a buffer object from a java double array, it fails with "TypeError: buffer object expected" and converting the array to a java string doesn't work neither.
Or If one tries to wrap a double[] in a DoubleBuffer, how do one convert this back to a ByteBuffer?
Would be nice, if you can provide some more detail.
from jpype.
I think that I misunderstood your comment about double arrays. I am basically reading the contents of a file into a byte array buf
, and then I use jpype to create a numpy uint8 array that I convert to whatever data type the original file contains. In other words, I am only using byte arrays in Java regardless of the data type contained in the file. This is what my code looks like when the byte array buf
contains the bytes from a file with little-endian 32-bit floats:
java_str = jpype._jclass.JClass('java.lang.String')(buf, 'ISO-8859-1')
arr = numpy.array(numpy.frombuffer(buffer(java_str.toString()),dtype='uint16'),dtype='uint8')
arr_float32 = numpy.frombuffer(buffer(arr),dtype='<f')
I am not sure if there is a fast way to convert a Java double array to a Java byte array. This would be a simple type cast in C, but in Java you may need to use a method like ByteBuffer.putDouble()
.
from jpype.
I believe your way of conversion may still be a bit faster than the code in #73, but not that hackish and so hoping you are willing to check it out š
from jpype.
What a nice improvement. On my machine it is around twice as fast as the string conversion hack, and it produces readable code!
from jpype.
will close this now, since the changes are merged
from jpype.
Related Issues (20)
- __eq__ for comparable should use equals()? HOT 5
- Intercept Java function calls HOT 3
- Access Java Enum .name() when a .name property is defined HOT 4
- JPype not implicity converting int to Java Integer for Java function call HOT 3
- use with Java making use of jogl, other native code. HOT 1
- Silent failure of jpype.startJVM when classpath is passed - RESOLVED HOT 4
- Load class error HOT 3
- No JVM shared library file (libjvm.so) found. Try setting up the JAVA_HOME environment variable properly. HOT 1
- Python 3.11 compatibility HOT 2
- CI runs only for JDK11? HOT 2
- JVM Crash when using a specific jar : cloudera HS2Driver (Hive), any versions HOT 2
- Failing to call a class that is compiled using JDK11 HOT 1
- No module named 'java.lang' HOT 1
- Segfault in JTerminate
- startJVM crashes Apple M1 HOT 7
- Rare crashes on interpreter exit HOT 1
- Support Python 3.12 HOT 7
- help(Class) doesn't work
- No module named 'java.lang' HOT 1
- startJVM throws EXCEPTION_ACCESS_VIOLATION on Windows 10 and 11 with jdk4py HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jpype.