schipholgroup / takeoff

Your runway to the cloud(s)!

Schiphol Takeoff

Takeoff: your runway to the cloud(s)!

This is Takeoff's documentation page, which provides a high-level overview of what Takeoff is, what is can do, and how you can use it in your environment.

Schiphol Takeoff is a deployment orchestration tool that abstract away much of the complexity of tying various cloud services together. Takeoff allows developers to focus on actual development work, without having to worry about coordinating a (large) number of cloud services to get things up and running across multiple environments. Takeoff itself is a Python package, that comes bundles in a Docker image. In this way, Takeoff is CI agnostic, assuming your CI provider allows running Docker containers.

Note

Takeoff is not aimed at deploying and/or configuring infrastructure. For tasks like this, consider using tools like Ansible, Puppet, or Terraform.

Principles

To understand better why Takeoff exists, it's important to mention some of the principles that were adhered to while building it:

1. Everything is Python (if possible. If it isn't, we use bash for some things). We chose Python for a number of reasons, but the most important ones are that Python code can easily be tested, has a lot of SDKs available for interacting with clouds and other external system, is easy to read, and is the lingua franca of data science (the initial target audience)
2. Modularity, modularity, and more modularity. By making the interactions with each external system a separate, decoupled module, we have freed the way for others to extend Takeoff to fulfill their requirements, without having to rewrite (too much) for every new component.
3. We ain't Ansible, Puppet, or Terraform. The goal of Takeoff was never, and will never, be to deploy infrastructure. Tools like Terraform are much better at this than we could ever hope to be. Takeoff is specifically aimed at your CI process when working with multiple cloud components, and having to orchestrate these components to work nicely together to deliver an awesome product.

Getting started

To get started quickly, have look here to read more on how to setup Takeoff in your project.

Contributing

To get started with contributing to Takeoff read the developers guide.

takeoff's Issues

Add docstrings to create_databricks_secrets_from_vault

Support uploading artifacts to DBFS

Allow artifact names to not be prefixed with a directory named after the application

Currently, the functions get_whl_name, get_jar_name and get_main_py_name return paths, not necessarily names. This means there is the assumption that everyone will have stored their wheels/jars/python files in a directly named after the application. While this should be a recommended practice, it shouldn't necessarily be hardcoded in Takeoff. Also, with this behaviour, the function names don't actually describe what the return value is; the return values are paths, not names.

Eventhub configuration assumes {dtap} should be appended to the end of the names of the eventhub entities

Given the following configuration

• task: createEventhubProducerPolicies
  policies:
  - eventhubEntity: sdhreclaimeventhub

Takeoff automatically adds the {dtap} to the end of the entity because of the following line and leaving no room to escape.

https://github.com/Schiphol-Hub/takeoff/blob/master/takeoff/azure/configure_eventhub.py#L166

Takeoff config / deloyment validator cli

create_databricks_secrets throws KeyError if no scopes exist

Using create_databricks_secrets, if you try to create a secret (and scope as well) when there are no scopes available at all yet, a KeyError is thrown. We should flag this to the user (rather than throw an error) and return False for _scope_exists instead.

Traceback (most recent call last):
File "/opt/miniconda3/bin/runway", line 4, in
import('pkg_resources').run_script('Runway==0.0.0', 'runway')
File "/opt/miniconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 661, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/opt/miniconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 1441, in run_script
exec(code, namespace, namespace)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/EGG-INFO/scripts/runway", line 4, in
main()
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/run_deployment.py", line 35, in main
run_task(env, task, task_config)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/run_deployment.py", line 44, in run_task
return deployment_steps[task](env, task_config).run()
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/create_eventhub_consumer_groups.py", line 65, in run
self.create_eventhub_consumer_groups(groups)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/create_eventhub_consumer_groups.py", line 209, in create_eventhub_consumer_groups
CreateDatabricksSecrets._create_scope(databricks_client, application_name)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/create_databricks_secrets.py", line 33, in _create_scope
if not CreateDatabricksSecrets._scope_exists(scopes, scope_name):
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/create_databricks_secrets.py", line 27, in scope_exists
return scope_name in set(["name"] for _ in scopes["scopes"])
KeyError: 'scopes'```

Deployments with name set in deployment config are not deleted as databricks job

Jobs are created, but takeoff does not find the job including the infix part in the databricks job deployment config.

Development branches all start a new job in databricks and result in several streams at once

When commenting from a development branch, the databricks job name contains the branch name. For streaming jobs in particular this is very annoying. Currently other jobs (committed from different branch) need to be stopped manually. One work around would be to set the application version plugin, which, following PR #68, would allow renaming the running job to a fix name. The issue then remains, that artifacts are also all named the same. These should be however clearly tied to a distinct name (either the branch name or any other set in the artifact plugin).
A proposed solution would be to create a databricks job name plugin.

Add docstrings to deploy_to_databricks

Drop file_extension parameter from artifact_name getters

The functions get_whl_name, get_jar_name, and get_main_py_name currently take file_extension as a parameter. Given the names of these functions, the file_extension parameter is pretty useless. It would be very odd for someone to try to get a jar that does not have .jar as extension (same for main_py or whl). We should therefore remove this parameter from these functions.

Links are broken in documentation - deployment steps

Interesting project! I am eager to try it out! However, some of the links are broken in the documentation, the example docker image, also the links in the deployment steps.

Allow batch Databricks jobs to be run immediately

Allow multiple Eventhub namespaces

At the moment the configure_eventhub task only allows interacting with a single Eventhub namespace. While this covers the majority of use cases, there are use cases in which you may have 1 project that interacts with multiple Eventhub namespaces. It would be nice if we could support that in Takeoff.

Allow option to prevent Databricks Job from being run after being submitted.

After inspecting takeoff/azure/deploy_to_databricks.py, there seems to be no option to do prevent the job from being run.

As I plan to schedule the job elsewhere, I don't need the job to be run after it has been submitted as a new DBJ.

Inconsistent base64 encoding during kubernetes deployment

During deploy_to_kubernetes, the deployment.yml configuration is rendered as a Jinja template. The variables that are available for substitution come from several places, including custom variables and Azure Keyvaults. The documentation currently states that variables from Azure Keyvaults are base64-encoded before substitution while variables from other sources are not.

The documentation, however, seems to be incorrect. Currently the behaviour seems to be that irrespective of source takeoff guesses whether the variable is already base64-encoded or not. If not, it gets encoded. If it is, it is left alone. This guessing is often wrong: it's essentially a heuristic and there are lots of non-encoded values that appear to be base64-encoded. (For example, the quick brown fox jumps over the lazy dog.)

Aside from the documentation mismatch, the impact of this is that variable interpolation is unpredictable and difficult to diagnose when issues go wrong. There is a workaround available: always base64-encode values in the Keyvault. (These will be left alone by takeoff and can be used as-is in the data section of a Secret, or decoded with a Jinja filter if the original value is required.)

Stepping back a bit, the purpose of this feature is to help with ensure that secrets stay so: values from the key vault should only be used in the Secret resource, which expects base64-encoded in the data attribute. This is laudable, but the current mechanism is error-prone.

Some ideas for resolving this issue:

• Update the implementation to match the documentation: values from the vault are unconditionally base64-encoded, other values are left alone.
• Leave all values alone. The user can then control base64 encoding (or not) using Jinja filters, which makes it easier to see what's going on and less magical. The user knows the context, and can decide appropriately what should be done.

From my perspective, I'd prefer the second option because it's more explicit. However it's most important that it behave predictably.

Rename parameters `env` to `application_version` everywhere

When deploying to k8s, wait for deployment to complete

At present, the deploy_to_kubernetes task is fire-and-forget: it submits the changes to the deployment, and if they apply the task succeeds.

For some resources, including Deployment changing the resource doesn't actually do the deployment, but instead triggers a bunch of things (rolling updates, etc.) It would be nice to have a way of blocking until that process either succeeds or fails. (When it fails for a Deployment, the changes made are rolled back automatically to the prior version of the resource, undoing the work of Takeoff.)

As a workaround at the moment, after invoking takeoff I add the following step:

• kubectl --namespace name-of-the-namesspace rollout status deployment/name-of-the-thing --watch=true

This piggy-backs on the k8s context that takeoff has set up (and authenticated) but is not ideal.

Databricks JOBs API key error

Given the fact that the Databricks instance is pretty much empty, we hit the similar bug with returning schema from Databricks.

Traceback (most recent call last):
File "/opt/miniconda3/bin/runway", line 4, in
import('pkg_resources').run_script('Runway==0.0.0', 'runway')
File "/opt/miniconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 661, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/opt/miniconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 1441, in run_script
exec(code, namespace, namespace)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/EGG-INFO/scripts/runway", line 4, in
main()
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/run_deployment.py", line 35, in main
run_task(env, task, task_config)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/run_deployment.py", line 44, in run_task
return deployment_steps[task](env, task_config).run()
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/deploy_to_databricks.py", line 59, in run
self.deploy_to_databricks(run_config)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/deploy_to_databricks.py", line 90, in deploy_to_databricks
self.__remove_job(databricks_client, app_name, self.env.artifact_tag, is_streaming=is_streaming)
File "/opt/miniconda3/lib/python3.7/site-packages/Runway-0.0.0-py3.7.egg/runway/deploy_to_databricks.py", line 150, in __remove_job
job_configs = [JobConfig(_["settings"]["name"], _["job_id"]) for _ in jobs_api.list_jobs()["jobs"]]
KeyError: 'jobs'

Building docker image for a release should also tag the image as `latest`

Extend current plugin model with steps

Context

I believe Takeoff would benefit a lot from allowing users to write their own deployment steps. @danielvdende asked me to write an issue to facilitate further discussion on this idea. Plugins would allow for broader use cases not envisioned by Takeoff's original creators without putting pressure on maintainers to add new steps. If community-built steps are mature enough they could be considered for upstream adoption. Similar plugin models already works well for projects like Apache Airflow and dbt.

Proposed solution

The easiest method is allowing users to write steps in the plugin-folder by inheriting from Step. Plugged steps could be listed in __init__.py or steps.py in the plugin-folder. The latter option would be consistent with Takeoff's own methodology. The method described here only supports local files and would not promote the use of community-built steps. One possible solution is to allow steps to be installed from PyPI or Github at runtime by specifying a package or repository.

Proposed plugin step functionality:

• Scan plugin folder for steps at runtime
• Steps can extend the base schema
• Plugin steps can be installed from remote sources (PyPI, Github?)
• Plugin steps can access credentials in Key Vault

Required work

• Enable use of plugin steps
• Write developer documentation for steps
• Maintain a list of community-provided steps

Let's discuss!