Example CI/CD with Databricks
Right now all dependencies, even the ones used for dev (e.g. chispa), are pinned and stored in requirements.txt
. This
is not optimal: at the very least prod and dev dependencies should be separated, and we should be able to know which
dependendencies are actually needed by our project and which dependencies are needed by those dependencies. In essence,
we should probably use a tool such as a pip-tools or
poetry.
We use pytest as our testing framework and chispa for testing Spark code.
We use black as our code formatter.
We configured it via file pyproject.toml
to use lines with at maximum 120 characters. NB: it's totally up to you and
your team to decide which length you want to use, either PEP's 79, 88, or another number. The nice thing about that is
that at least we show how we can configure black.
To check which files would be formatted with which modification (without actually performing the modifications), run:
black --diff --color .
To format the code:
black .
If you can't remember that command, simply run:
make format
NB: black is used as a check in the CI.
We use flake8 to check for code compliance with (most of) PEP8
rules. It is configured with file .flake8
.
To check for PEP8 compliance:
flake8 .
If you can't remember that command, simply run:
make check
NB: flake8 is used as a check in the CI.
- use proper .gitignore file
- create virtual env
- install dependencies and pin them in requirements.txt
Those scripts use various Python libraries that need to be installed first. So before running them, create a virtual environment with Python 3.7+ and run:
pip install -r ./cd-scripts/requirements.txt
We use the python script ./cd-scripts/create_prod_job.py
to create the job. Make sure module requests
is installed
in your (virtual) environment.
Run:
python ./cd-scripts/create_job.py \
--databricks-instance <databricks-instance> \
--pat <pat> \
--wheel-version <wheel-version> \
--dbfs-wheel-dir <dbfs-wheel-dir> \
--scope <scope> \
--storage-uri <storage-uri> \
--storage-key <storage-key>
This command will output the associated job_id
(that you will have to reuse to update the job).
Run:
python ./cd-scripts/update_job.py \
--job-id <job-id> \
--databricks-instance <databricks-instance> \
--pat <pat> \
--wheel-version <wheel-version> \
--dbfs-wheel-dir <dbfs-wheel-dir> \
--scope <scope> \
--storage-uri <storage-uri> \
--storage-key <storage-key>