setting up in-memory data analytics for qflex about cloudsuite

[ustiugov]

Hi,

Current suggestion is to use cloudsuite w/o Docker in QFlex. The solution with docker will be released but much later.

Regards,
Dmitrii

[Jeongseob]

Hi Dmitrii,

Thanks for letting me know. I was wondering the documentation is still accessible somewhere. At the very beginning, Cloudsuite released the documents as well. That's why I was finding the documents.

Thanks,
Jeongseob

[ustiugov]

You may want to take a look at the Dockerfiles we advertise for our workloads.

[Jeongseob]

Hi Dmitrii,

While I am looking at the Dockerfile, cloudsuite/spark, I am seeing the SPARK setup environment. One thing I would like to ask you is that where the SPARK related environment variables are declared such as SPARK_HOME.

Here is the container image from the docker file.

root@57eec44d7c15:/# env | grep SPARK
SPARK_HOME=/opt/spark-2.1.0
SPARK_VERSION=2.1.0
SPARK_MASTER_PORT=7077
SPARK_WEBUI_PORT=8080
root@57eec44d7c15:/#

Although the environmental variables have been defined somewhere, I couldn't find the location.

[ZiluTian]

Hi Jeongseob,

Please take a look at the Dockerfile for spark-base here, which defines the SPARK_HOME and some other variables
https://github.com/parsa-epfl/cloudsuite/blob/master/benchmarks/spark-base/spark-base/Dockerfile

You might also find these files helpful:
https://github.com/parsa-epfl/cloudsuite/tree/master/benchmarks/spark-base

Please let me know if you have further questions.

Best,
Zilu

[Jeongseob]

It helps me a lot. Thank Zilu.