Question: Is it somehow possible to use a Context and execute statements in parallel? about apache-spark-node

[henridf]

Good question. I can't think of a good way to do this with the current implementation.

That said, this is something I'd been thinking about as one of the possible next enhancements to the library... concurrency needs to be supported and made easy by this library. An obvious use case would be a node rest api server using this library to start/manager spark jobs. (As an aside, I'd be curious to know your use case if you can share).

Created issues #26 and #27 for this work.

[tobilg]

Actually, I have created a REST API server which is working quite nicely. Unfortunately I can't OpenSource it currently because it was created for my employer. But I'm planning on disussing this, and hopefully find a solution.

As of the node-java docs I tend to think it's impossible to leverage Java threads for this purpose, see

• https://github.com/joeferner/node-java#javaNewProxy

I did some tests, and even when using the Java thread classes the calls are still synchronous. This is aus well documented in the link above.

I couldn't find another solution yet unfortunately.

[henridf]

I saw your question on node-java and looked into javaNewProxy but I think that proxies are a red herring here - maybe due to the unfortunate choice of Thread in the example of the node-java docs.

My guess is that it should be possible to leverage threads (following something like joeferner/node-java#290 (comment)). If you can share your tests where you used the Java thread classes but still had synchronous calls, that would be very helpful.

[tobilg]

Maybe I misunderstood this... But I don't think Joe's answer really solves my problem, because the thread would need to run the JS Spark code, and I don't see how this can work within a Java class.

I'll try to make some Gists out of my trials, but this will be difficult because I used my own abstractions of the Spark context.

Furthermore, I don't really see how interfaces can be implemented with node-java other than with new Proxy().

There's another implementation of node-java which can apparently do this, see https://github.com/kyriosli/node-java#implement but the project seems to be not as mature as the other.

var vm = require('./node-java').createVm();
var Runnable = vm.implement(['java/lang/Runnable'], {
    'run()V' : function() {
        console.log('Thread started');
       // Spark code goes here
    }
});

var thread = vm.findClass('java/lang/Thread').newInstance('Ljava/lang/Runnable;', Runnable.newInstance());
thread.invoke('start()V');

[tobilg]

Regarding the REST API Server, I'm in discussions with my colleagues and superiors. Hopefully I can reach a decision next week.

[tobilg]

Fixed in #27