fivestars-os / aladdin

Magical command line tool for kubernetes cluster management and application development

Aladdin

What is Aladdin?

Inspired by the Genie of the Lamp, Aladdin is a command line tool built by Fivestars to simplify kubernetes operations for clusters management and application development. We have done this by combining several command line tools including kops, k3d, docker, kubectl, helm, awscli, and git.

Use aladdin to:

• Create and manage an aws kubernetes cluster
• Upgrade your aws kubernetes cluster
• Run and deploy your organization's applications across different environments (including locally) with environment specific configuration
• Run operation-type commands against your application (e.g. putting your application into maintenance mode)

How does Aladdin work?

Aladdin has two main components. One component runs on your host machine, and one component runs in a docker container, started by the first component.

The host component is responsible for:

• Parsing command line options
• Running any commands to be executed on the host machine
• Checking and warning about any missing dependencies or installing the dependency in ~/.aladdin/bin if possible
• Starting k3d (make sure your docker engine is running before running aladdin!)
• Pulling the aladdin image
• Running the aladdin docker container (the container component)

The container component is responsible for

• Running commands to manage your aws kubernetes cluster
• Running commands to deploy and manipulate your applications on your kubernetes cluster

Script Diagram/Architecture

Installing Aladdin

To set up, just clone the Aladdin GitHub repository to get started:

$ git clone [email protected]:fivestars-os/aladdin.git
$ cd aladdin
$ scripts/infra_k8s_check.sh

The infra_k8s_check.sh script checks to see if all of aladdin's dependencies are installed. Depending on the dependency, it will warn if it is missing or install it in ~/.aladdin/bin if possible. This script is also run every time you run aladdin.

The recommended way to install aladdin is using the install script:

$ ./scripts/install-aladdin.sh

This script will install pipx and then use it to install aladdin in its own virtual environment.

You may also want to add ~/.aladdin/bin to your path:

$ export PATH=$PATH:~/.aladdin/bin

You can add these two commands to your profile file if desired.

You will now need to create your aladdin configuration, and link that to aladdin.

$ aladdin config set config_repo [email protected]:{git_account}/{repo}.git

NOTE:

When doing bootstrapping, initial setup or when developing on the aladdin codebase itself it might be useful to be able to point aladdin to a config on the local development environment (ie not yet in source control). This can be done by setting the ALADDIN_DEV=true environment variable and by setting the config_dir setting (aladdin config set config_dir /path/to/your/config).

Manage configuration

Software dependencies

aladdin installs software it needs like helm, jq, aws cli etc.

But you may not need that if this applies to your situation:

1. You already have a separate process for config management of your machine.
2. You want to keep using your tools (scripts, ansible playbook etc).
3. You don't want aladdin to install any software.

Then run:

aladdin config set manage.software_dependencies false

k3d

aladdin uses k3d to support local development

Local Cluster Resource Configuration

Currently, you can configure cpu, memory, and disk by going to your docker desktop UI, going to preferences, and then going to resources. The recommended configuration is 2 CPUs, 8GB memory, and 60GB disk.

Creating and managing an aws kubernetes cluster

This is all encapsulated in the aladdin cluster command

Integrating your application with aladdin

There is some plumbing you need to add to your application source code in order to get it to integrate with aladdin. We have created an aladdin-demo project that walks you through these steps. It also includes what we think of as "best practices" in developing applications on kubernetes. You can also use this project to verify your aladdin installation is working correctly. Please refer to the aladdin demo instructions here.

Project development

We have several aladdin commands used for development and deployment. Note that are you implicitly or explicitly calling these commands with respect to a namespace and cluster (through the -n and -c flags).

Aladdin local dev commands

• aladdin build used to build a project's docker images locally with the local tag
• aladdin start used to install or update a project locally
• aladdin stop used to uninstall a project locally
• aladdin restart used to uninstall and reinstall a project locally
• aladdin publish used to publish your project's docker images to your ecr and publish your helm packages to s3 to be used to deploy to non local environments

Aladdin non local commands

• aladdin deploy used to deploy to a remote, non-local cluster
• aladdin undeploy used to remove a project from a remote, non-local cluster
• aladdin rollback used to go back to a previous deployment

Aladdin local and non local commands

• aladdin cmd used to issue commands against a project (e.g. putting a project into maintenance)
• aladdin environment used to get or maniuplate config maps
• aladdin refresh issue a no-op deployment, which is useful to restart all pods in a deployment
• aladdin scale used to change the number of replicas for a deployment
• aladdin connect used to connect to a container's shell

Sample development workflow

• git clone [email protected]:{git account name}/{project repo name}.git
• cd {project repo name}
• git checkout -b {feature branch}
• aladdin build
• aladdin start
• do some development
• commit code and push to remote
• aladdin publish (publishes current hash, remember what that hash is)
• aladdin -c {remote integration cluster} -n {your personal namespace} deploy {project-repo-name} {hash from previous step} - Your app will be running at {service-name}.{namespace}.{cluster root dns} or {service-name}.{service_dns_suffix} if you specified a service_dns_suffix
• Test to make sure everything is working right
• Merge your feature branch to master
• Pull latest master, and run aladdin publish. Remeber the hash it is building off of.
• Deploy to your production cluster aladdin -c {production cluster} deploy {project-repo-name} {hash from previous step}
• Your app will be running at {service-name}.{namespace}.{cluster root dns} or {service-name}.{service_dns_suffix} if you specified a service_dns_suffix
• Rollback if necessary with aladdin -c {production cluster} rollback {project-repo-name}

Other useful aladdin commands

• aladdin bash drop into the aladdin container bash, with your context set to your current cluster and namespace, with all container commands aliased.
• aladdin clean remove all stopped and unused docker images.
• aladdin config maniuplate various aladdin config.
• aladdin create-namespace create the namespace you pass in.
• aladdin delete-namespace delete the namespace you pass in after removing all the helm packages on that namespace.
• aladdin get-dashboard-url will output the url of your cluster's dashboard assuming it is installed.
• aladdin host give instructions to update your local /etc/hosts file for k3d ingress compatibility.
• For a complete list of aladdin commands, run aladdin -h.

Optional arguments to aladdin

• -h/--help show help.
• -c/--cluster which cluster to connect to, defaults to LOCAL.
• -n/--namespace which namespace to connect to, defaults to default.
• --init force initialization logic (i.e. pull latest aladdin image, test aws config, initialize helm, etc...). This is forced every hour for each cluster/namespace combo.
• --skip-prompts skip any confirmation messages during aladdin execution. Useful when automating commands.
• --non-terminal run aladdin container without tty.

Additionally aladdin checks for a ALADDIN_DEV=true environment variable that will enable aladdin development options/features such as:

• mounting the host's aladdin directory onto the aladdin container. This means code changes should be reflected on the container without the need to re-build the image. Mostly useful when developing aladdin.
• specifying an alternate aladdin docker image using the ALADDIN_IMAGE environment variable

Running several aladdin commands in the same cluster/namespace combo

Aladdin supports running several commands in the same cluster/namespace combo without having to "reinitialize" aladdin. To do this, go into aladdin bash. Then all the container commands will be aliased to be run without prefixing aladdin. Example:

$ aladdin bash
     . . .
Launching bash shell. Press CTRL+D to come back to this menu.
This bash contain a lot of functions to help
Don't forget to checkout scripts/bash_profile.bash in aladdin
LOCAL:default> build
Building aladdin-demo docker image (~30 seconds)
docker build -t aladdin-demo:local -f app/Dockerfile .
Sending build context to Docker daemon  28.16kB
     . . .
Successfully tagged aladdin-demo-commands:local
LOCAL:default> start
INFO:Found cluster values file
     . . .
LOCAL:default> refresh aladdin-demo-server
INFO:Refreshing deployment aladdin-demo-server
LOCAL:default>

Tests

Right now we have some e2e tests for aladdin that come in two flavors: aladdin test-local and aladdin test-remote. These tests require some configuration, an example of which can be found here. You will need to make some modifications to this config so it has access to create and destroy a temporary cluster on your aws account. Then:

aladdin config set config_dir /path/to/your/e2e/tests/config
aladdin test-local
aladdin test-remote

Adding tests is a great way to get started with contributing to aladdin!

Plugins

Aladdin also has the ability to invoke user plugins. The directory structure must be as follows:

plugins-folder-name/
  container/
    plugin-name-1/
      plugin-name-1 (must match directory name)
      plugin-name-1-helper-1
      plugin-name-1-helper-2
    plugin-name-2/
      plugin-name-2 (must match directory name)
    plugin-name-3/
      plugin-name-3 (must match directory name)
      plugin-name-3-helper-1
  host/
    plugin-name-4/
      plugin-name-4 (must match directory name)
      plugin-name-4-helper-1
      plugin-name-4-helper-2
    plugin-name-5/
      plugin-name-5 (must match directory name)
    plugin-name-6/
      plugin-name-6 (must match directory name)
      plugin-name-6-helper-1

Then:

aladdin config set plugin_dir /path/to/plugins-folder-name`
aladdin plugin-name-2 # Execute plugin 2
aladdin plugin-name-5 # Execute plugin 5

Plugins are useful for adding lightweight helper functions that are personally useful. If you can't fit it into this model, consider extending aladdin by creating a custom aladdin docker image, and updating your config to use that image. If you think your plugin is universally useful, consider creating a PR and adding it to aladdin itself.

Other Features

Ingress per namespace feature

Aladdin supports an ingress per namespace feature. This is off by default. We recommend using this for your shared development cluster to keep the number of elbs low. To turn this on, you'll need to do the following steps:

• Pull our open sourced ingress-nginx and alter the values files to your organization's needs. Then, use aladdin to publish it to your ecr.
• Add this to your cluster's config.json file:

    "namespace_init": [
        {"project": "ingress-nginx", "ref": "<your published ref>"}
    ],
    "ingress_info": {
        "use_ingress_per_namespace": true,
        "ingress_controller_service_name": "ingress-nginx",
        "ingress_name": "ingress"
    }

The "namespace_init" field tells aladdin to install the ingress-nginx project on namespace creation. This will be needed on remote clusters, but not on LOCAL, since k3d comes with that out of the box. The "ingress_info" field tells aladdin how to sync your ingress. Services installed on a cluster with this feature will want to have their service type set to NodePort rather than LoadBalancer. This is most easily done by setting it in the values.yaml in your cluster's directory in aladdin config, i.e. adding this:

  service:
    publicServiceType: "NodePort"

and then referencing {{ service.publicServiceType }} in your service yaml file.

Troubleshooting

Here is a list of common issues and their solutions.

aladdin's Issues

[BUG] AUTHENTICATION_ENABLED: unbound variable

If kops validate cluster doesn't succeed (either because the cluster is not ready, or if there is a kube-system pod that is in a failed state, we won't call _handle_authentication_config(), which means our AUTHENTICATION_ENABLED variable won't get set, which means we can't call aladdin bash to interact with the cluster. Instead we get:

/root/aladdin/commands/bash/container/bash/bash: line 12: AUTHENTICATION_ENABLED: unbound variable

Bug: `ALADDIN_CONFIG_DIR: unbound variable` error on v1.19.7.36

On a fresh setup, while attempting to install and setup aladdin according to README instructions

NOTE:

When doing bootstrapping, initial setup or when developing on the aladdin codebase itself it might be useful to be able to point aladdin to a config on the local development environment (ie not yet in source control). This can be done by setting the ALADDIN_DEV=true environment variable and by setting the config_dir setting (aladdin config set config_dir /path/to/your/config).

... I got the following result

➜  aladdin git:(main) export ALADDIN_DEV=true
➜  aladdin git:(main) ./scripts/install-aladdin.sh 
Installing to existing venv 'aladdin'
  installed package aladdin 1.19.7.36, installed using Python 3.9.6
  These apps are now globally available
    - aladdin
done! ✨ 🌟 ✨
➜  aladdin git:(main) aladdin config set config_dir ~/code/aladdin-config/
/Users/<...removed...>/site-packages/scripts/shared.sh: line 11: ALADDIN_CONFIG_DIR: unbound variable

Troubleshooting

Some light debugging didn't reveal anything obvious, so I rolled the version back one version, which seemed to work

➜  aladdin git:(main) export ALADDIN_VERSION=1.19.7.35                    
➜  aladdin git:(main) ./scripts/install-aladdin.sh    
Installing to existing venv 'aladdin'
  installed package aladdin 1.19.7.35, installed using Python 3.9.6
  These apps are now globally available
    - aladdin
done! ✨ 🌟 ✨
➜  aladdin git:(main) aladdin config set config_dir ~/code/aladdin-config/
➜  aladdin git:(main) 

Possible cause

I don't have much insight into the codebase yet, but this change to the script's execution order stands out to me as something which might result in this type of bug.

Steps to reproduce

$ cd aladdin
$ git pull
$ export ALADDIN_DEV=true
$ export ALADDIN_VERSION=1.19.7.36
$ ./scripts/install-aladdin.sh
$ aladdin config set config_dir /path/to/local/aladdin-config/

Allow configure of default local cluster

Aladdin users can now use different local clusters (docker-desktop, k3s etc) instead of just the minikube.

The default value of local cluster is set to minikube in the code. Need a way to set a different local cluster value in aladdin config so that user can type just aladdin bash for example instead of aladdin -c docker-desktop bash for example.

Allow cluster name with dashes

When I use cluster name like docker-desktop, it fails with

jq: error: desktop/0 is not defined at <top-level>, line 1:
.cluster_aliases.docker-desktop
jq: 1 compile error

Add support for Helm v3

Helm v3 has been released, aladdin should support both helm v2 and v3 projects, my current proposal is to add a:

{
    "helm_version": "2" # or "3"
}

into lamp.json that instructs aladdin what major version of helm to use per project, additionally this could also be set at the cluster config level, to set the default helm version to use for a cluster, rather than specify each project individually

More info: https://helm.sh/docs/topics/v2_v3_migration/

Allow option to skip installing software

Aladdin installs minikube, jq etc

But if I have another solution in place to manage configuration of my machine (like ansible playbook for example) I would continue to use that and still use aladdin for rest of the use cases.

Since current behavior is to install software. We can make it the default option.

Something like INSTALL_SOFTWARE=true and users can override with INSTALL_SOFTWARE=false when they invoke aladdin if they choose to.

[BUG] aladdin connect command fails when there are hanging evicted pods

When minikube evicts pods, they sometimes end up hanging around in minikube and they don't show up in kubectl get pods (but they do show up with kubectl get po -a --all-namespaces). This causes aladdin connect to error out when it gets to the evicted pods.

minikube:default> connect cpay

Available:
----------
0: pod cpay-6c6f68d8dc-rk6np; container cpay
Traceback (most recent call last):
  File "/root/aladdin/commands/python/main.py", line 94, in <module>
    args.func(args)
  File "/root/aladdin/commands/python/command/connect.py", line 14, in connect_args
    connect(args.app, args.namespace)
  File "/root/aladdin/commands/python/command/connect.py", line 32, in connect
    for container in pod.status.container_statuses:
TypeError: 'NoneType' object is not iterable

This situation can be resolved by manually force deleting those pods with the following command, but it is not clear for a user why the connect command would be erroring out.

kubectl get po -a --all-namespaces -o json | \
jq  '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) |
"kubectl delete po \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c

Expected behavior:
aladdin should ignore evicted pods and only try to list out connect options for active pods.

Update doc and code to clarify our usage of 'minikube'

In the code we use minikube to imply local development plus source of docker.

But with recent PR #67 - we are making management of minikube optional when user might have installed docker with alternate means.

So we have to revisit our code and docs to clarify why minikube is used in each of the places: Is it for dependencies like docker? or is it to deploy helm chart for local development?

Also need to consider possibility that cluster code is minikube but manage.minikube is false. In that case we may have to allow overriding of that config setting.

Allow option to ask aladdin to not manage my minikube instance

Aladdin sets up minikube in a certain way:

1. Set up nfs mounting,
2. Set config based on config set in aladdin with minikube.key style etc.

If I already have a way to manage minikube, I would prefer to do it using my existing setup and prefer to use aladdin for rest of the use cases.

Perhaps an argument like --no-manage-minikube? That way, if that flag isn't passed, aladdin can proceed with the current/default behavior.

Allow network host option with aladdin docker container

When using solutions like kind (kubernetes in docker) - the cluster is accessible from local ip like https://127.0.0.1:64354.

So to access it from inside aladdin docker container, need to run docker in host network.

Since this setting is needed only for some clusters and not others, we could consider this setting to be included in cluster specific settings like in aladdin-config/kind-kind/config.json -- where we already have is_local, similarly we can have use_host_network and set it true only for kind-kind config.

We can then implement default of use_host_network false if not present/set.

Add support for Google Kubernetes Engine

A lot of teams use Google Kubernetes Engine (GKE) to host their K8s cluster, are there any plans to support GKE? How much work would it take to add support for GKE?

--set-override-values only works at the end

deploy --set-override-values abc=abc project hash won't work because project and hash are consumed by the --set-override-values

Handle truncated list of route 53 record sets

We can get this on deploys if there are too many hosted zones:

Traceback (most recent call last):
  File "/root/aladdin/commands/python/main.py", line 94, in <module>
    args.func(args)
  File "/root/aladdin/commands/python/command/deploy.py", line 35, in deploy_args
    args.repo, args.set_override_values)
  File "/root/aladdin/commands/python/command/deploy.py", line 76, in deploy
    sync_dns.sync_dns(namespace)
  File "/root/aladdin/commands/python/command/sync_dns.py", line 28, in sync_dns
    cr.dual_dns_prefix_annotation_name, cr.ingress_info))
  File "/root/aladdin/commands/python/cluster_rules.py", line 96, in fill_hostedzone
    check_ns_values(self._boto, main_dns_id, self.sub_dns, dns_ns)
  File "/root/aladdin/commands/python/libs/aws/dns_mapping.py", line 92, in check_ns_values
    raise Exception('Does not handle truncated hostedzone')
Exception: Does not handle truncated hostedzone

Namespace creation fails on long namespace names

We are hitting an aws limit on idempotencyToken for the request certificate call. We should either spit out a better error, or supply an idempotencyToken to the requestCertificate call. This will increase our domain name limit to 64.

aladdin get-dashboard-url doesn't work on local minikube

Need to use minikube dashboard instead