zhztheplayer / arrow-1 Goto Github PK

Hi there @zhztheplayer,

I am currently using your nice Arrow Dataset Java implementation to be able to use Datasets from Java. I can use it without problems on my own system, where I have sudo rights, have up-to-date gcc, cmake, etcetera.

However, I cannot run my jarfile, containing all dependencies, on a remote machine.
Here is the error I get on a remote machine:

Exception in thread "Thread-49" java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.dataset.file.JniWrapper
	at org.apache.arrow.dataset.file.FileSystemDatasetFactory.createNative(FileSystemDatasetFactory.java:33)
	at org.apache.arrow.dataset.file.FileSystemDatasetFactory.<init>(FileSystemDatasetFactory.java:29)
	at arrow.dataset.DefaultDatasetFactory.lambda$create$0(DefaultDatasetFactory.java:49)
	at arrow.JNIMemory.withAllocatorDo(JNIMemory.scala:54)
	at arrow.dataset.DefaultDatasetFactory.create(DefaultDatasetFactory.java:48)
	at arrow.dataset.MetaDataset.dataset$lzycompute(MetaDataset.scala:34)
	at arrow.dataset.MetaDataset.dataset(MetaDataset.scala:34)

I want to stress again that the very same jar file works without problems on my local machine.

I suppose the error makes sense: I access the JniWrapper here for the very first time, so this is the first time the '.so' shared libraries are called right?

My local system has a lot more recent libraries than the server I ran on while getting this error.
My guess: Different environments cause the .so file to fail loading propely, meaning I can use my code only on the same system I compiled this JNI bridge on.

I noticed there was a way to compile static libraries ('.a' files) in Arrow. That might solve the problem. With '.a' files, relevant system library routines are inserted into the '.a' file,
so my server environment does not matter much then (I think).

Questions

Here is what I would like to know:

1. Do you think using static libraries will solve my problem?
2. Does your JNI bridge currently support static libraries?
3. Do you plan to add static libraries to your JNI bridge in the future?

If static libraries prove to increase independence from system libraries, I really think you should use those instead of shared libraries. Even though the library size will increase, static libs would add a piece of compatibility between systems, and Java and JARs are all about compatibility between systems.

P.S.
JNI can use static libraries since Java 8. Source: StackOverflow