Integrate spline with pyspark about spline

[brunoRenzo6]

@vackosar @wajda SPLINE is a greate tool, we've been using it in our company inside jobs but just like @maorshechter we work with pyspark to run our process, therefore we came up with a solution for SPLINE and PYSPARK Integration.
The solution is : write a PysparkHandler object in Scala to receive the SparkSession from PySpark, and return a SparkSession with SPLINE enabled.
Pretty Simple Implementation, take a look : https://github.com/WilliamPorto/keyruslab-spline/blob/master/Integra%C3%A7%C3%A3o.md

[wajda]

We never think about it, but sounds like a good idea. I'll work on it.

[wajda]

https://stackoverflow.com/questions/44082957/how-to-add-a-sparklistener-from-pyspark-in-python
and
http://aseigneurin.github.io/2016/09/01/spark-calling-scala-code-from-pyspark.html

[maorshechter]

The thread on stackoverflow looks very complicated , is it the simplest way? I wish I could use it with easy clone and plug&play .. if I implement the spark listener ( your second sulotion, is it enough to make spline working ( I would like to see full example , will help a lot)

…

On Oct 18, 2018 5:00 PM, "Alex Vayda" ***@***.***> wrote: https://stackoverflow.com/questions/44082957/how-to-add- a-sparklistener-from-pyspark-in-python — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#48 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWEhXwmASLIhBGa7G5UuCvMLWmIH0qDYks5umImDgaJpZM4WX90r> .

[vackosar]

Have u been able to resolve this. I recon the solution wud be also akin to Java use:
#43 (comment)

[wajda]

@brunoRenzo6 Wow, that's awesome! Thank you for contribution! I'll add it to the Spline docs.

[vackosar]

@brunoRenzo6 thanks for the tips. I modified the our API and here is example how you can run Spline via PySpark: https://github.com/AbsaOSS/spline/blob/release/0.3/sample/src/main/scala/za/co/absa/spline/sample/batch/python_sample.py

It should be released soon as part of 0.3