cordon-thiago / airflow-spark

Docker with Airflow and Spark standalone cluster

Airflow Spark

This project contains the following containers:

• postgres: Postgres database for Airflow metadata and a Test database to test whatever you want.

  • Image: postgres:9.6
  • Database Port: 5432
  • References: https://hub.docker.com/_/postgres

• airflow-webserver: Airflow webserver and Scheduler.

  • Image: docker-airflow-spark:1.10.7_3.1.2
  • Port: 8282

• spark: Spark Master.

  • Image: bitnami/spark:3.1.2
  • Port: 8181
  • References:
    • https://github.com/bitnami/bitnami-docker-spark
    • https://hub.docker.com/r/bitnami/spark/tags/?page=1&ordering=last_updated

• spark-worker-N: Spark workers. You can add workers copying the containers and changing the container name inside the docker-compose.yml file.

  • Image: bitnami/spark:3.1.2
  • References:
    • https://github.com/bitnami/bitnami-docker-spark
    • https://hub.docker.com/r/bitnami/spark/tags/?page=1&ordering=last_updated

• jupyter-spark: Jupyter notebook with pyspark for interactive development.

  • Image: jupyter/pyspark-notebook:spark-3.1.2
  • Port: 8888
  • References:
    • https://hub.docker.com/layers/jupyter/pyspark-notebook/spark-3.1.2/images/sha256-37398efc9e51f868e0e1fde8e93df67bae0f9c77d3d3ce7fe3830faeb47afe4d?context=explore
    • https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pyspark-notebook
    • https://hub.docker.com/r/jupyter/pyspark-notebook/tags/

Architecture components

Setup

Clone project

$ git clone https://github.com/cordon-thiago/airflow-spark

Build airflow Docker

Inside the airflow-spark/docker/docker-airflow

$ docker build --rm --force-rm -t docker-airflow-spark:1.10.7_3.1.2 .

Optionally, you can override the arguments in the build to choose specific Spark, Hadoop and Airflow versions. As an example, here is how to build an image containing Airflow version 1.10.14, Spark version 2.4.7 and Hadoop version 2.7.

$ docker build --rm --force-rm \
-t docker-airflow-spark:1.10.14_2.4.7 . \
--build-arg AIRFLOW_VERSION=1.10.14 \
--build-arg SPARK_VERSION=2.4.7 \
--build-arg HADOOP_VERSION=2.7

Spark and hadoop versions follow the versions as defined at Spark download page: https://spark.apache.org/downloads.html

Airflow versions can be found here: https://pypi.org/project/apache-airflow/#history

If you change the name or the tag of the docker image when building, remember to update the name/tag in docker-compose file.

Start containers

Navigate to airflow-spark/docker and:

$ docker-compose up

If you want to run in background:

$ docker-compose up -d

Note: when running the docker-compose for the first time, the images postgres:9.6, bitnami/spark:3.1.2 and jupyter/pyspark-notebook:spark-3.1.2 will be downloaded before the containers started.

Check if you can access

Airflow: http://localhost:8282

Spark Master: http://localhost:8181

PostgreSql - Database Test:

• Server: localhost:5432
• Database: test
• User: test
• Password: postgres

Postgres - Database airflow:

• Server: localhost:5432
• Database: airflow
• User: airflow
• Password: airflow

Jupyter Notebook: http://127.0.0.1:8888

• For Jupyter notebook, you must copy the URL with the token generated when the container is started and paste in your browser. The URL with the token can be taken from container logs using:

  $ docker logs -f docker_jupyter-spark_1
  

How to run a DAG to test

1. Configure spark connection acessing airflow web UI http://localhost:8282 and going to Connections

2. Edit the spark_default connection inserting spark://spark in Host field and Port 7077

3. Run the spark-test DAG

4. Check the DAG log for the task spark_job. You will see the result printed in the log

5. Check the spark application in the Spark Master web UI (http://localhost:8181)

How to run the Spark Apps via spark-submit

After started your docker containers, run the command below in your terminal:

$ docker exec -it docker_spark_1 spark-submit --master spark://spark:7077 <spark_app_path> [optional]<list_of_app_args>

Example running the hellop-world.py application:

$ docker exec -it docker_spark_1 spark-submit --master spark://spark:7077 /usr/local/spark/app/hello-world.py /usr/local/spark/resources/data/airflow.cfg

Increasing the number of Spark Workers

You can increase the number of Spark workers just adding new services based on bitnami/spark:3.1.2 image to the docker-compose.yml file like following:

spark-worker-n:
        image: bitnami/spark:3.1.2
        user: root
        networks:
            - default_net
        environment:
            - SPARK_MODE=worker
            - SPARK_MASTER_URL=spark://spark:7077
            - SPARK_WORKER_MEMORY=1G
            - SPARK_WORKER_CORES=1
            - SPARK_RPC_AUTHENTICATION_ENABLED=no
            - SPARK_RPC_ENCRYPTION_ENABLED=no
            - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
            - SPARK_SSL_ENABLED=no
        volumes:
            - ../spark/app:/usr/local/spark/app # Spark scripts folder (Must be the same path in airflow and Spark Cluster)
            - ../spark/resources/data:/usr/local/spark/resources/data #Data folder (Must be the same path in airflow and Spark Cluster)

Adding Airflow Extra packages

Rebuild Dockerfile (in this example, adding GCP extra):

$ docker build --rm --build-arg AIRFLOW_DEPS="gcp" -t docker-airflow-spark:1.10.7_3.1.2 .

After successfully built, run docker-compose to start container:

$ docker-compose up

More info at: https://github.com/puckel/docker-airflow#build

Useful docker commands

List Images:
$ docker images <repository_name>

List Containers:
$ docker container ls

Check container logs:
$ docker logs -f <container_name>

To build a Dockerfile after changing sth (run inside directoty containing Dockerfile):
$ docker build --rm -t <tag_name> .

Access container bash:
$ docker exec -i -t <container_name> /bin/bash

Useful docker-compose commands

Start Containers:
$ docker-compose -f <compose-file.yml> up -d

Stop Containers:
$ docker-compose -f <compose-file.yml> down --remove-orphans

Extras

Spark + Postgres sample

• The DAG spark-postgres.py loads movies.csv and ratings.csv data into Postgres tables and query these tables to generate a list of top 10 movies with more rates.
  • This DAG runs the load-postgres.py and read-postgres.py applications. These applications are also available in the notebooks load-postgres-notebook.ipynb and read-postgres-notebook.ipynb.

airflow-spark's Issues

Why pulling images when using *build*?

Usually, images are pulled by running docker build, so why pulling the three images again?

ERROR: manifest for jupyter/pyspark-notebook:3.0.1 not found:

Bringing up the structure on Ubuntu 20.04.2 LTS fails with:

ERROR: manifest for jupyter/pyspark-notebook:3.0.1 not found:

ModuleNotFoundError: No module named 'wtforms.compat'

Hi!

I'm keep getting this error while doing docker-compose up:

airflow-webserver_1 | ModuleNotFoundError: No module named 'wtforms.compat'

I've followed all the steps from the .md file. Any clue on what could be happening?

btw I'm running it on WSL2.

build docker-airflow-spark:2.7.0_3.4.1

Hey when i run

docker build --rm --force-rm \ -t docker-airflow-spark:2.7.0_3.4.1 . \ --build-arg AIRFLOW_VERSION=2.7.0 \ --build-arg SPARK_VERSION=3.4.1 \ --build-arg HADOOP_VERSION=3

It produces an error

airflow-webserver_1 | /entrypoint.sh: line 78: airflow: command not found airflow-webserver_1 | /entrypoint.sh: line 83: exec: airflow: cannot execute: Is a directory airflow-webserver_1 | /entrypoint.sh: line 81: airflow: command not found

Spark and all others are running but the airflow-webserver not.

Error when try to save spark dataframe to .csv file

try 1:
df.write.format('csv').mode('overwrite').save(.'/home/my.csv', header=True, index=False, sep='|')
try 2:
df.coalesce(1).write.option('mode', 'append').option('header', 'true').csv("./home/my.csv")
try 3:
spark_df.repartition(1).write.format('com.databricks.spark.csv').save("./tmp/data.csv", header='true', index=False, sep='|')

Don't save any dataframe to a csv file, always show the same error (see below)

python version

I want to use python 3.8, which tag should I use since I can see from dockerfile that stretch is preferred？But I can't seem to find a python3.8-stretch version on docker hub.

ERROR: failed to solve: process "/bin/sh -c set -ex ... && pip freeze" did not complete successfully: exit code: 100

Hi all! On Windows 10 in any console (cmd, git bash) after

docker build --rm -t docker-airflow2:latest .

i got the error:

ERROR: failed to solve: process "/bin/sh -c set -ex && buildDeps=' freetds-dev libkrb5-dev libsasl2-dev libssl-dev libffi-dev libpq-dev git ' && apt-get update -yqq && apt-get upgrade -yqq && apt-get install -yqq --no-install-recommends $buildDeps freetds-bin build-essential default-libmysqlclient-dev apt-utils curl rsync netcat locales iputils-ping telnet && locale-gen && update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 && useradd -ms /bin/bash -d ${AIRFLOW_HOME} airflow && pip install -U pip setuptools wheel && pip install pytz && pip install pyOpenSSL && pip install ndg-httpsclient && pip install pyasn1 && pip install -r requirements.txt && pip install --use-feature=2020-resolver apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh${AIRFLOW_DEPS:+,}${AIRFLOW_DEPS}]==${AIRFLOW_VERSION} && pip install 'redis>=2.10.5,<3' && if [ -n \"${PYTHON_DEPS}\" ]; then pip install ${PYTHON_DEPS}; fi && apt-get purge --auto-remove -yqq $buildDeps && apt-get autoremove -yqq --purge && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/man /usr/share/doc /usr/share/doc-base && python --version && pip freeze" did not complete successfully: exit code: 100

Docker seems to be workable.

What's wrong?

Cannot edit spark_default connection

I cannot seem to edit the spark_default from the Airflow UI. Is there any way that this can be done?
I keep seeing this,

                          ____/ (  (    )   )  \___
                         /( (  (  )   _    ))  )   )\
                       ((     (   )(    )  )   (   )  )
                     ((/  ( _(   )   (   _) ) (  () )  )
                    ( (  ( (_)   ((    (   )  .((_ ) .  )_
                   ( (  )    (      (  )    )   ) . ) (   )
                  (  (   (  (   ) (  _  ( _) ).  ) . ) ) ( )
                  ( (  (   ) (  )   (  ))     ) _)(   )  )  )
                 ( (  ( \ ) (    (_  ( ) ( )  )   ) )  )) ( )
                  (  (   (  (   (_ ( ) ( _    )  ) (  )  )   )
                 ( (  ( (  (  )     (_  )  ) )  _)   ) _( ( )
                  ((  (   )(    (     _    )   _) _(_ (  (_ )
                   (_((__(_(__(( ( ( |  ) ) ) )_))__))_)___)
                   ((__)        \\||lll|l||///          \_))
                            (   /(/ (  )  ) )\   )
                          (    ( ( ( | | ) ) )\   )
                           (   /(| / ( )) ) ) )) )
                         (     ( ((((_(|)_)))))     )
                          (      ||\(|(|)|/||     )
                        (        |(||(||)||||        )
                          (     //|/l|||)|\\ \     )
                        (/ / //  /|//||||\\  \ \  \ _)
-------------------------------------------------------------------------------
Node: 5fce0b10ba4b
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line 2138, in edit_view
    form = self.edit_form(obj=model)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line 1340, in edit_form
    return self._edit_form_class(get_form_data(), obj=obj)
  File "/usr/local/lib/python3.6/site-packages/wtforms/form.py", line 208, in __call__
    return type.__call__(cls, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_admin/form/__init__.py", line 16, in __init__
    super(BaseForm, self).__init__(formdata=formdata, obj=obj, prefix=prefix, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/wtforms/form.py", line 274, in __init__
    self.process(formdata, obj, data=data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/wtforms/form.py", line 126, in process
    if obj is not None and hasattr(obj, name):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 356, in __get__
    retval = self.descriptor.__get__(instance, owner)
  File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", line 212, in get_extra
    return fernet.decrypt(bytes(self._extra, 'utf-8')).decode()
  File "/usr/local/lib/python3.6/site-packages/cryptography/fernet.py", line 199, in decrypt
    raise InvalidToken
cryptography.fernet.InvalidToken

It shoot an error while build building Airflow webserver i.e /usr/bin/env: ‘bash\r’: No such file or directory

Adding data base connections

This repository is great, thanks.

In order to extend the usability, a database connection to an external PostgreSQL DB can be added, too.

spark_job SparkSubmitOperator failed

Hey and thanks for the nice project,

when I follow the steps from the README it ends up with an Error:

[2020-11-02 09:33:37,559] {{taskinstance.py:1088}} ERROR - Cannot execute: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']. Error code is: 1.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 955, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 181, in execute
self._hook.submit(self._application)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 362, in submit
spark_submit_cmd, returncode
airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']. Error code is: 1.
[2020-11-02 09:33:37,560] {{taskinstance.py:1117}} INFO - All retries failed; marking task as FAILED
[2020-11-02 09:33:40,835] {{logging_mixin.py:112}} INFO - [2020-11-02 09:33:40,834] {{local_task_job.py:103}} INFO - Task exited with return code 1`

Seems
log.txt

ERROR: The image for the service you're trying to recreate has been removed

when execute docker-compose up -d get this error:

Pulling airflow-webserver (docker-airflow-spark:1.10.7_3.1.2)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

airflow-webserver_1 | ModuleNotFoundError: No module named 'wtforms.compat'

To solve it I added to the requirements.txt:

wtforms==2.3.3

A newer version of wtforms like 3.0.0 doesn't work!

How can I add Kafka to this build?

Thank you.

Adding new Postgres DB

Hello Friends
Can someone elaborate on how to add another Postgres DB using this Docker? I have been trying without success.
I am really admiring how easy it is to install a cluster ready to play with.
Would be awesome to use a DB dedicated strictly to my self-studying project instead of sharing it with Airflow. Thanks in advance.

IMO below does not works:

PostgreSql - Database Test:

Server: localhost:5432
Database: test
User: test
Password: postgres

I have been playing around with docker/docker-airflow files to make it work.

cryptography.fernet.InvalidToken

I define the DB connection in pySpark. So no need for an Airflow-Connection. Why there is an error like the one below?

[2021-03-02 08:00:14,932] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: etl_sad_reporting.etl_customer_dialog 2021-03-02T07:58:57.850780+00:00 [running]> fee10b35f03f
[2021-03-02 08:00:14,966] {{taskinstance.py:1088}} ERROR -
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 955, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 179, in execute
spark_binary=self._spark_binary
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 149, in init
self._connection = self._resolve_connection()
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 183, in _resolve_connection
conn = self.get_connection(self._conn_id)
File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 84, in get_connection
log.info("Using connection to: %s", conn.log_info())
File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", line 321, in log_info
"XXXXXXXX" if self.extra_dejson else None))
File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", line 338, in extra_dejson
if self.extra:
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 356, in get
retval = self.descriptor.get(instance, owner)
File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", line 212, in get_extra
return fernet.decrypt(bytes(self._extra, 'utf-8')).decode()
File "/usr/local/lib/python3.6/site-packages/cryptography/fernet.py", line 194, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
[2021-03-02 08:00:14,969] {{taskinstance.py:1117}} INFO - All retries failed; marking task as FAILED
[2021-03-02 08:00:24,812] {{logging_mixin.py:112}} INFO - [2021-03-02 08:00:24,812] {{local_task_job.py:103}} INFO - Task exited with return code 1

very slow build of docker-airflow-spark:1.10.7_3.1.2

Hi there,

I'm running:
$ docker build --rm --force-rm -t docker-airflow-spark:1.10.7_3.1.2 .

and it's taking a really, really long time - it's held up on downloading the cryptography requirement, and is pulling down different versions, incredibly slowly. It looks related to this:
https://stackoverflow.com/questions/65122957/resolving-new-pip-backtracking-runtime-issue

any ideas?

thanks

[+] Building 10649.5s (6/16)                                                                                                                  
 => => extracting sha256:71c316554a558d4aa95c0be17c01e4a0366d6026a98eeba53dca5baa6091e815                                                0.2s 
 => => sha256:8973a3f2f69f7267196f90924c19cb72348d9b41f9c0a1bb9bc11180eaa3ac0c 14.21MB / 14.21MB                                        39.7s 
 => => sha256:1f96a121555fcc5208421bb785d99b8fb9a32bd0615e1e7854295600e7319873 241B / 241B                                              37.2s
 => => extracting sha256:721081de66bfc648ce19234c6333d6e031344f6ac90904476c9ec2dba2917e3a                                                2.5s
 => => sha256:19593799e97757b3b713045650eb23af9fc6cfa0303ddc51f0fc31cea074c3fc 2.21MB / 2.21MB                                          38.3s
 => => extracting sha256:239fb482263d7865a4a15a52e4a4be892393343c0c4318a930f770d6da32cba0                                                7.9s
 => => extracting sha256:ec2643d613a16bf8b8eb10a67654b8b4afa12ebc14be6deca2c6e8c9935a1197                                                0.3s
 => => extracting sha256:8973a3f2f69f7267196f90924c19cb72348d9b41f9c0a1bb9bc11180eaa3ac0c                                                0.6s
 => => extracting sha256:1f96a121555fcc5208421bb785d99b8fb9a32bd0615e1e7854295600e7319873                                                0.0s
 => => extracting sha256:19593799e97757b3b713045650eb23af9fc6cfa0303ddc51f0fc31cea074c3fc                                                0.2s
 => [ 2/12] COPY requirements.txt /requirements.txt                                                                                      1.3s 
 => [ 3/12] RUN set -ex     && buildDeps='         freetds-dev         libkrb5-dev         libsasl2-dev         libssl-dev           10559.9s 
 => => # ld take a while.                                                                                                                     
 => => # Collecting cryptography>=0.9.3                                                                                                       
 => => #   Using cached cryptography-3.4.8-cp36-abi3-manylinux_2_24_x86_64.whl (3.0 MB)                                                       
 => => #   Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)                                                         
 => => #   Downloading cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)                                                         
 => => #   Downloading cryptography-3.4.5-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)      

Different Spark Versions

After cloning and bringing it up, the version of the spark cluster is 3.0.1 and not the same as the spark version 2.4.5 and the spark_test job fails.

Its running after adding spark 3.0.1.

The Log for the failed spark_test under Spark 2.4.5 here:

Reading local file: /usr/local/airflow/logs/spark-test/spark_job/2020-12-02T15:48:06.551084+00:00/2.log
[2020-12-02 15:50:15,406] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: spark-test.spark_job 2020-12-02T15:48:06.551084+00:00 [queued]>
[2020-12-02 15:50:15,416] {{taskinstance.py:655}} INFO - Dependencies all met for <TaskInstance: spark-test.spark_job 2020-12-02T15:48:06.551084+00:00 [queued]>
[2020-12-02 15:50:15,416] {{taskinstance.py:866}} INFO -

[2020-12-02 15:50:15,416] {{taskinstance.py:867}} INFO - Starting attempt 2 of 2
[2020-12-02 15:50:15,416] {{taskinstance.py:868}} INFO -

[2020-12-02 15:50:15,426] {{taskinstance.py:887}} INFO - Executing <Task(SparkSubmitOperator): spark_job> on 2020-12-02T15:48:06.551084+00:00
[2020-12-02 15:50:15,427] {{standard_task_runner.py:52}} INFO - Started process 2010 to run task
[2020-12-02 15:50:15,448] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,448] {{dagbag.py:403}} INFO - Filling up the DagBag from /usr/local/airflow/dags/spark-test.py
[2020-12-02 15:50:15,460] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: spark-test.spark_job 2020-12-02T15:48:06.551084+00:00 [running]> 0f1bfb481b22
[2020-12-02 15:50:15,498] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,497] {{base_hook.py:84}} INFO - Using connection to: id: spark_default. Host: spark://spark, Port: 7077, Schema: None, Login: None, Password: None, extra: XXXXXXXX
[2020-12-02 15:50:15,499] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,498] {{spark_submit_hook.py:305}} INFO - Spark-Submit cmd: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']
[2020-12-02 15:50:15,843] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,842] {{spark_submit_hook.py:436}} INFO - Using properties file: null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,877] {{spark_submit_hook.py:436}} INFO - Parsed arguments:
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - master spark://spark:7077
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - deployMode null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - executorMemory null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - executorCores null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - totalExecutorCores null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - propertiesFile null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverMemory null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverCores null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverExtraClassPath null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverExtraLibraryPath null
[2020-12-02 15:50:15,878] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,878] {{spark_submit_hook.py:436}} INFO - driverExtraJavaOptions null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - supervise false
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - queue root.default
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - numExecutors null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - files null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - pyFiles null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - archives null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - mainClass null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - primaryResource file:/usr/local/spark/app/hello-world.py
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - name Spark Hello World
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - childArgs [/usr/local/spark/resources/data/airflow.cfg]
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - jars null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - packages null
[2020-12-02 15:50:15,879] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,879] {{spark_submit_hook.py:436}} INFO - packagesExclusions null
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - repositories null
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - verbose true
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - Spark properties used, including those specified through
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - --conf and those from the properties file null:
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO - (spark.master,spark://spark:7077)
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:15,880] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:15,880] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,094] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[2020-12-02 15:50:16,458] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,458] {{spark_submit_hook.py:436}} INFO - Main class:
[2020-12-02 15:50:16,458] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,458] {{spark_submit_hook.py:436}} INFO - org.apache.spark.deploy.PythonRunner
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,458] {{spark_submit_hook.py:436}} INFO - Arguments:
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - file:/usr/local/spark/app/hello-world.py
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - null
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - /usr/local/spark/resources/data/airflow.cfg
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - Spark config:
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - (spark.master,spark://spark:7077)
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - (spark.app.name,Spark Hello World)
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - (spark.submit.deployMode,client)
[2020-12-02 15:50:16,459] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO - Classpath elements:
[2020-12-02 15:50:16,460] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,459] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,460] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,460] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,460] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,460] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:50:16,697] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,697] {{spark_submit_hook.py:436}} INFO - Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
[2020-12-02 15:50:16,698] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,698] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkContext: Running Spark version 2.4.5
[2020-12-02 15:50:16,709] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,709] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkContext: Submitted application: Spark Hello World
[2020-12-02 15:50:16,733] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,733] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing view acls to: airflow
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,733] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing modify acls to: airflow
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,734] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing view acls groups to:
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,734] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: Changing modify acls groups to:
[2020-12-02 15:50:16,734] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,734] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(airflow); groups with view permissions: Set(); users with modify permissions: Set(airflow); groups with modify permissions: Set()
[2020-12-02 15:50:16,899] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,899] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO Utils: Successfully started service 'sparkDriver' on port 35829.
[2020-12-02 15:50:16,911] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,911] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkEnv: Registering MapOutputTracker
[2020-12-02 15:50:16,919] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,919] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkEnv: Registering BlockManagerMaster
[2020-12-02 15:50:16,921] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,921] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
[2020-12-02 15:50:16,921] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,921] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
[2020-12-02 15:50:16,925] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,925] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-6d01f882-8c44-46db-a2ba-fc7c78b245fb
[2020-12-02 15:50:16,933] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,933] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
[2020-12-02 15:50:16,939] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:16,939] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:16 INFO SparkEnv: Registering OutputCommitCoordinator
[2020-12-02 15:50:17,022] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,022] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO Utils: Successfully started service 'SparkUI' on port 4040.
[2020-12-02 15:50:17,046] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,046] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://0f1bfb481b22:4040
[2020-12-02 15:50:17,093] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,093] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2020-12-02 15:50:17,120] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:17,120] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:17 INFO TransportClientFactory: Successfully created connection to spark/172.26.0.4:7077 after 16 ms (0 ms spent in bootstraps)
[2020-12-02 15:50:37,094] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:37,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:37 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2020-12-02 15:50:57,094] {{logging_mixin.py:112}} INFO - [2020-12-02 15:50:57,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:50:57 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2020-12-02 15:51:17,095] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,094] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
[2020-12-02 15:51:17,095] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,095] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 WARN StandaloneSchedulerBackend: Application ID is not initialized yet.
[2020-12-02 15:51:17,099] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,099] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41935.
[2020-12-02 15:51:17,099] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,099] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO NettyBlockTransferService: Server created on 0f1bfb481b22:41935
[2020-12-02 15:51:17,100] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,100] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO SparkUI: Stopped Spark web UI at http://0f1bfb481b22:4040
[2020-12-02 15:51:17,100] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,100] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
[2020-12-02 15:51:17,102] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,102] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO StandaloneSchedulerBackend: Shutting down all executors
[2020-12-02 15:51:17,104] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,104] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
[2020-12-02 15:51:17,106] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,106] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 WARN StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
[2020-12-02 15:51:17,108] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,108] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
[2020-12-02 15:51:17,115] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,115] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,117] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,117] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMasterEndpoint: Registering block manager 0f1bfb481b22:41935 with 366.3 MB RAM, BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,119] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,119] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,119] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,119] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 0f1bfb481b22, 41935, None)
[2020-12-02 15:51:17,125] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,124] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO MemoryStore: MemoryStore cleared
[2020-12-02 15:51:17,125] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,125] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManager: BlockManager stopped
[2020-12-02 15:51:17,128] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,128] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO BlockManagerMaster: BlockManagerMaster stopped
[2020-12-02 15:51:17,129] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,129] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
[2020-12-02 15:51:17,144] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,144] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO SparkContext: Successfully stopped SparkContext
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 ERROR SparkContext: Error initializing SparkContext.
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - at scala.Predef$.require(Predef.scala:224)
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
[2020-12-02 15:51:17,191] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,191] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.SparkContext.(SparkContext.scala:516)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.Gateway.invoke(Gateway.java:238)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - at java.lang.Thread.run(Thread.java:748)
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO SparkContext: SparkContext already stopped.
[2020-12-02 15:51:17,192] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,192] {{spark_submit_hook.py:436}} INFO - Traceback (most recent call last):
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/spark/app/hello-world.py", line 14, in
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - sc = SparkContext()
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 136, in init
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - conf, jsc, profiler_cls)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 198, in _do_init
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - self._jsc = jsc or self._initialize_context(self._conf._jconf)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/pyspark/context.py", line 306, in _initialize_context
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - return self._jvm.JavaSparkContext(jconf)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1525, in call
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - answer, self._gateway_client, None, self._fqn)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - File "/usr/local/lib/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - format(target_id, ".", name), value)
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
[2020-12-02 15:51:17,193] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,193] {{spark_submit_hook.py:436}} INFO - : java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at scala.Predef$.require(Predef.scala:224)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.SparkContext.(SparkContext.scala:516)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.Gateway.invoke(Gateway.java:238)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
[2020-12-02 15:51:17,194] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
[2020-12-02 15:51:17,195] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,194] {{spark_submit_hook.py:436}} INFO - at py4j.GatewayConnection.run(GatewayConnection.java:238)
[2020-12-02 15:51:17,195] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,195] {{spark_submit_hook.py:436}} INFO - at java.lang.Thread.run(Thread.java:748)
[2020-12-02 15:51:17,195] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,195] {{spark_submit_hook.py:436}} INFO -
[2020-12-02 15:51:17,214] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,214] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO ShutdownHookManager: Shutdown hook called
[2020-12-02 15:51:17,214] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,214] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-5dd50760-7c6a-4173-83d1-944b6d7d7858
[2020-12-02 15:51:17,215] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:17,215] {{spark_submit_hook.py:436}} INFO - 20/12/02 15:51:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-d9d497e5-4635-4b2c-b057-ba01ca569db7
[2020-12-02 15:51:17,546] {{taskinstance.py:1088}} ERROR - Cannot execute: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']. Error code is: 1.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 955, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/operators/spark_submit_operator.py", line 181, in execute
self._hook.submit(self._application)
File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/spark_submit_hook.py", line 362, in submit
spark_submit_cmd, returncode
airflow.exceptions.AirflowException: Cannot execute: ['spark-submit', '--master', 'spark://spark:7077', '--conf', 'spark.master=spark://spark:7077', '--name', 'Spark Hello World', '--verbose', '--queue', 'root.default', '/usr/local/spark/app/hello-world.py', '/usr/local/spark/resources/data/airflow.cfg']. Error code is: 1.
[2020-12-02 15:51:17,547] {{taskinstance.py:1117}} INFO - All retries failed; marking task as FAILED
[2020-12-02 15:51:20,567] {{logging_mixin.py:112}} INFO - [2020-12-02 15:51:20,566] {{local_task_job.py:103}} INFO - Task exited with return code 1

airflow Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Cant work, i run dag spark submit and get spark writes that there are no resources
How to fix it?

[2022-05-26 13:09:06,603] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,603] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO MemoryStore: MemoryStore started with capacity 1458.6 MiB
[2022-05-26 13:09:06,616] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,616] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO SparkEnv: Registering OutputCommitCoordinator
[2022-05-26 13:09:06,766] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,766] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO Utils: Successfully started service 'SparkUI' on port 4040.
[2022-05-26 13:09:06,813] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,813] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://01c927d91284:4040
[2022-05-26 13:09:06,826] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,826] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO SparkContext: Added JAR file:///usr/local/spark/resources/jars/postgresql-42.3.6.jar at spark://01c927d91284:35835/jars/postgresql-42.3.6.jar with timestamp 1653570546285
[2022-05-26 13:09:06,827] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,827] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO SparkContext: Added JAR file:/usr/local/spark/resources/jars/ss-user-rule-algorithm-1.0.jar at spark://01c927d91284:35835/jars/ss-user-rule-algorithm-1.0.jar with timestamp 1653570546285
[2022-05-26 13:09:06,972] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:06,972] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:06 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark:7077...
[2022-05-26 13:09:07,005] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,005] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO TransportClientFactory: Successfully created connection to spark/172.19.0.6:7077 after 19 ms (0 ms spent in bootstraps)
[2022-05-26 13:09:07,071] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,071] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20220526130907-0034
[2022-05-26 13:09:07,073] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,072] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220526130907-0034/0 on worker-20220526111530-172.19.0.4-42347 (172.19.0.4:42347) with 2 core(s)
[2022-05-26 13:09:07,074] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,074] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneSchedulerBackend: Granted executor ID app-20220526130907-0034/0 on hostPort 172.19.0.4:42347 with 2 core(s), 1024.0 MiB RAM
[2022-05-26 13:09:07,075] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,074] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20220526130907-0034/1 on worker-20220526111530-172.19.0.4-42347 (172.19.0.4:42347) with 2 core(s)
[2022-05-26 13:09:07,075] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,075] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneSchedulerBackend: Granted executor ID app-20220526130907-0034/1 on hostPort 172.19.0.4:42347 with 2 core(s), 1024.0 MiB RAM
[2022-05-26 13:09:07,079] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,078] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33001.
[2022-05-26 13:09:07,079] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,079] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO NettyBlockTransferService: Server created on 01c927d91284:33001
[2022-05-26 13:09:07,080] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,080] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
[2022-05-26 13:09:07,086] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,086] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220526130907-0034/0 is now RUNNING
[2022-05-26 13:09:07,087] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,087] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20220526130907-0034/1 is now RUNNING
[2022-05-26 13:09:07,087] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,087] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 01c927d91284, 33001, None)
[2022-05-26 13:09:07,090] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,090] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO BlockManagerMasterEndpoint: Registering block manager 01c927d91284:33001 with 1458.6 MiB RAM, BlockManagerId(driver, 01c927d91284, 33001, None)
[2022-05-26 13:09:07,092] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,092] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 01c927d91284, 33001, None)
[2022-05-26 13:09:07,093] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,093] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 01c927d91284, 33001, None)
[2022-05-26 13:09:07,265] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:07,265] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:07 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
[2022-05-26 13:09:25,992] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:25,992] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
[2022-05-26 13:09:40,994] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:40,993] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
[2022-05-26 13:09:55,993] {{logging_mixin.py:112}} INFO - [2022-05-26 13:09:55,993] {{spark_submit_hook.py:436}} INFO - 22/05/26 13:09:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

How to add a new DAG

How to add a new DAG?
After a rebuild of Dockerfile and restarting the Container I was able to find a DAG under DAG/ catalogue.
However, I still can't see it under Airflow UI
Thanks for help

Update

is it possible to update docker to new versions?

An error occurred while calling o74.save.

Hey @cordon-thiago,

running the postgres example produces the following:

File "/usr/local/spark/app/read-postgres.py", line 72, in
df_result.coalesce(1).write.format("csv").mode("overwrite").save("/usr/local/spark/resources/data/output_postgres", header=True)
py4j.protocol.Py4JJavaError: An error occurred while calling o74.save.

new version of pip

Hey,

when building the docker image there appears this ERROR message.

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

marshmallow-sqlalchemy 0.24.0 requires marshmallow>=3.0.0, but you'll have marshmallow 2.21.0 which is incompatible.

Use spark 3.0.2 as most repos do not have 3.0.1 anymore

Spark 3.0.1 is no longer available and building jupyter/pyspark-notebook generates the following error

❯ docker build --rm --force-rm -t jupyter/pyspark-notebook:3.0.1 .
[+] Building 3.3s (7/12)
 => [internal] load build definition from Dockerfile                                                                                                     0.0s
 => => transferring dockerfile: 3.27kB                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                          0.0s
 => [internal] load metadata for docker.io/jupyter/scipy-notebook:latest                                                                                 2.7s
 => [1/9] FROM docker.io/jupyter/scipy-notebook@sha256:00af391facb071b6b6191893555811a7680a3d7c40eb1bac5c17540e6131f625                                  0.0s
 => CACHED [2/9] RUN apt-get -y update &&     apt-get install --no-install-recommends -y     "openjdk-11-jre-headless"     ca-certificates-java &&       0.0s
 => CACHED [3/9] WORKDIR /tmp                                                                                                                            0.0s
 => ERROR [4/9] RUN wget -q $(python -c "import requests; content = requests.get('https://www.apache.org/dyn/closer.lua/spark/spark-3.0.1/spark-3.0.1-b  0.5s
------
 > [4/9] RUN wget -q $(python -c "import requests; content = requests.get('https://www.apache.org/dyn/closer.lua/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz?as_json').json(); print(content['preferred']+content['path_info'])"):
------
executor failed running [/bin/bash -o pipefail -c wget -q $(python -c "import requests; content = requests.get('https://www.apache.org/dyn/closer.lua/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz?as_json').json(); print(content['preferred']+content['path_info'])")]: exit code: 8

Updating dockerfile's spark versio and checksum values fixes this issue
Old

ARG spark_version="3.0.1"
--
ARG spark_checksum="F4A10BAEC5B8FF1841F10651CAC2C4AA39C162D3029CA180A9749149E6060805B5B5DDF9287B4AA321434810172F8CC0534943AC005531BB48B6622FBE228DDC"

New

ARG spark_version="3.0.2"
--
ARG spark_checksum="a9bd16d6957579bb2f539d88f83ef5a5005bfbf2909078691397f0f1590b6a0e73c7fd6d51a0b1d69251a1c4c20b9490006b8fa26ebe37b87e9c0cee98aa3338"

Error on Step 2: Build Jupyter docker

`
=> ERROR [4/9] RUN wget -q $(wget -qO- https://www.apache.org/dyn/closer.lua/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz?as_json | python -c "impo 1.1s

[4/9] RUN wget -q $(wget -qO- https://www.apache.org/dyn/closer.lua/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz?as_json | python -c "import sys, json; content=json.load(sys.stdin); print(content['preferred']+content['path_info'])") && echo "F4A10BAEC5B8FF1841F10651CAC2C4AA39C162D3029CA180A9749149E6060805B5B5DDF9287B4AA321434810172F8CC0534943AC005531BB48B6622FBE228DDC *spark-3.0.1-bin-hadoop2.7.tgz" | sha512sum -c - && tar xzf "spark-3.0.1-bin-hadoop2.7.tgz" -C /usr/local --owner root --group root --no-same-owner && rm "spark-3.0.1-bin-hadoop2.7.tgz":

---

executor failed running [/bin/bash -o pipefail -c wget -q $(wget -qO- https://www.apache.org/dyn/closer.lua/spark/spark-${APACHE_SPARK_VERSION}/spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz?as_json | python -c "import sys, json; content=json.load(sys.stdin); print(content['preferred']+content['path_info'])") && echo "${spark_checksum} *spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" | sha512sum -c - && tar xzf "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" -C /usr/local --owner root --group root --no-same-owner && rm "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz"]: exit code: 8
`