Problem running mist on own Apache Spark Cluster about mist

[leonid133]

Hi @akshungupta, thanks for the issue. Could you please share settings for your Apache Spark cluster and docker.conf?

[akshungupta]

Hi,
Thank you for getting back to me.

The settings for Apache Spark are as follows:

1. I have 1 master and 2 worker nodes each on different machines. All of the machines can talk to each other.
2. I have used sbin/start-master.sh and sbin/start-slave.sh scripts (part of Apache Spark installation) to start the masters and slaves (workers).
3. There are no other custom settings that I have configured to create my Apache Spark cluster. It is a very simple cluster that I have brought up consisting of three machines.

I have attached my docker.conf, router.conf and the application simple_streaming.py in configs_job.zip to this post. simple_streaming.py is a very simple application and we wanted to get a feel of deploying mist on Apache Spark cluster.

configs_job.zip

Here is how I compiled the mist image and ran it:

1. sudo docker build -t test_mist .
2. sudo docker run -p 2003:2003 --name mist -v /var/run/docker.sock:/var/run/docker.sock -v $PWD/configs:/usr/share/mist/configs -v $PWD/jobs:/jobs test_mist mist

One thing to note is the following:

We had downloaded the source code of Apache Spark and had used the binary from the project to launch the Apache Spark cluster and tell mist the IP and port of the master. We could not find a way to use mist's apache spark binary to set up a custom cluster. Please also let us know if this is possible.

[leonid133]

Hi @akshungupta

You can start workers and connect them to the master via:

./sbin/start-slave.sh <master-spark-URL>

If you used spark-defaults.conf show me content please.

Generally it is better to use the IP address of Master node instead of the alphabetic hostname that is provided by spark master.

spark://<master-machine-IP>:7077

Once you have started a worker, look at the master’s web UI (port 8080 or 4040 by default). You should see all your nodes listed there.

[akshungupta]

Hi @leonid133

I had started my workers and master using sbin/start-mater.sh and sbin/start-slave.sh scripts available as part of the Spark code base. I did see all my worker nodes on the master's web UI.

I am not sure whether this error is because I am not using mist's own spark binary or not. I am using a different binary but same spark version (2.1.0).

I did not use spark-defaults.conf. I just started 2 slaves on 2 different machines and provided the master spark url to each slave when running the script /sbin/start-slave.sh

I tried using the IP address instead of the DNS name but I am getting the same error.

[leonid133]

Hi @akshungupta

Try to check the connection from docker container to spark master node and SPARK_VERSION

docker exec -it <mist_docker_name> bash
telnet <master_host> <master_port>
echo $SPARK_VERSION

Now for Spark version 2.1.0 we have a specific docker

docker run -p 2003: 2003 -v /var/run/docker.sock:/var/run/docker.sock -d hydrosphere / mist: master-2.1.0 Mist

[akshungupta]

Hi @leonid133, using master-2.1.0 fixed the issue!

Thank you!