I'm sorry for opening so many issues...
But I am using the swarm version of HDFS and Spark, in particular I think it is a version of few weeks ago. This is the docker-compose-spark.yml file:
version: '3.2'
services:
spark-master:
image: vzzarr/spark-master:gatk_env
networks:
- core
deploy:
replicas: 1
mode: replicated
restart_policy:
condition: on-failure
ports:
- 8080:8080
- 7077:7077
env_file:
- ./hadoop-swarm.env
volumes:
- /data0/reference/hg19-ucsc/:/reference/hg19-ucsc/
- /data0/fastq/:/fastq/
- ../../:/NGS-SparkGATK/
- /data/ngs/:/ngs/
spark-worker:
image: bde2020/spark-worker:2.1.0-hadoop2.8-hive-java8
networks:
- core
environment:
- SPARK_MASTER=spark://spark-master:7077
deploy:
replicas: 5
#mode: global
restart_policy:
condition: on-failure
env_file:
- ./hadoop-swarm.env
networks:
core:
external: true
and how you can see there are some personal modifies to your original file, because of implementation necessity.
In particular I am using two Azure VM for testing, where one is the Swarm Master and the other a Swam Worker.
I would expected that all the Spark Worker were directly connected to the Spark Master, in a similar manner of how explained in this Spark tutorial. But as reported in that Spark Cluster tutorial: "Once you have started a worker, look at the master’s web UI (http://localhost:8080 by default). You should see the new node listed there, along with its number of CPUs and memory (minus one gigabyte left for the OS)".
But using lynx http://localhost:8080
(lynx is a terminal browser) while I am executing a Spark Job, I can see:
* URL: spark://475ac7c0cd02:7077
* REST URL: spark://475ac7c0cd02:6066 (cluster mode)
* Alive Workers: 0
* Cores in use: 0 Total, 0 Used
* Memory in use: 0.0 B Total, 0.0 B Used
* Applications: 0 Running, 0 Completed
* Drivers: 0 Running, 0 Completed
* Status: ALIVE
It seems to me that the Spark Master can't see the Spark Workers. Moreover, while I can see that the Swarm Master is really "working" (20-30 GB of RAM occupied and all the CPUs working at 97-100%), the Swarm Worker load of work is clearly lower (<2 GB of RAM and CPUs at 0-2%), giving me the impression that the work load is not distributed.
So my question is: is it handled the worker connection to the master? Or I should provide it (as they suggest in the Spark tutorial with ./sbin/start-slave.sh <master-spark-URL>
)? Or I am doing something wrong?
Thank you very much for your patience and your time.