Improvement proposal about jgit-spark-connector

[eiso]

I really like this idea. And I think it's closer to what @mcuadros was thinking of as well. /cc @marnovo @campoy

[marnovo]

Yes, looks more straightforward.

Two questions regarding the comparison to the current:

from sourced.engine import Engine
from pyspark.sql import SparkSession
from pyspark.sql.functions import *

spark = SparkSession.builder \
        .master("local[*]").appName("Examples") \
        .getOrCreate()

engine = Engine(spark, "/repositories")

1. In practical terms, what is lost in terms of flexibility from the current approach?
2. Is it more likely that abstracting this away from the user might break something under certain circumstances or make it harder to debug in case of a problem?

[ajnavarro]

Is not possible to do the proposed API changes on the engine because, when you start a spark-shell, notebook, and so on, the spark session is provided. Because of that, we wouldn't create a new session inside the Engine wrapper, we should use the existing one.

For the other hand, use standard python console is another of the several ways that we could run the engine (JVM application, scala-shell, pyspark-shell, jupyter scala and python notebooks, zeppelin notebooks, and so on.)

If it's really important support the standard python shell right now, we can do that, but I think we already have a lot of methods to use the engine in a really simple way.

add pyspark to PYTHONPATH during installation

I think this is a really bad practice. Change environment variables during a dependency download? I don't think that is even possible using pipy.

[eiso]

@ajnavarro I think your argument makes sense. Closing this issue now.