Can this be used as a Apache Spark Node.js API or will there only be a shell? about apache-spark-node

[henridf]

This project is all pretty raw and early on, but it already exposes (some of) the Spark APIs (see README.md).

So you could definitely access the Spark dataframe APIs via this. I'll try to add some examples soon, but in the meantime the simplest way would be to mimic the relevant bits of bin/spark-node (in particular, create a sqlContext object by calling spark.sqlContext(..).

[bf]

Thanks @henridf for your fast response.

Do you think it is feasible to implement the wordcount.py script using the current API (with some fiddling of course)? I am currently calling spark-submit from node..

It goes like this:

    sc = SparkContext(appName="PythonWordCount")
    lines = sc.textFile(sys.argv[1], 1)
    counts = lines.flatMap(lambda x: x.split(' ')) \
                  .map(lambda x: (x, 1)) \
                  .reduceByKey(add)
    output = counts.collect()
    for (word, count) in output:
        print("%s: %i" % (word, count))
    sc.stop()

[henridf]

Yes! It is possible to do word count with the current API. Check out the example that I just added to the README.

You'll notice that it uses the DataFrame APIs rather than the RDD APIs like wordcount.py, which is preferable (where that is possible).

[bf]

Wow thank you very much, this is great! I will try to contribute to this project over the next few weeks as I incorporate spark into my current project.

[henridf]

Cool! Keep the questions/feedback coming as you make progress.