Would be nice to have this in Datawire's Homebrew tap for easy installation on macOS

It would be really nice to be able to add custom code before and after commands. A real world use case of this would be to do something like istioctl kube-inject after forge build ... or forge deploy .... However, other use cases would be to glue in custom code that goes off and does something like send a Slack message or make an API call to another service.

Spit balling a bit of implementation I would like to see this work similar to how cookiecutter handles hooks: There is a directory called hooks which can have an arbitrary executable programs dropped into it. Some consideration will need to make, however, to account for the different commands in forge and also that there is likely some amount of ordering that needs to be followed.

It's not clear what I am supposed to do here... and I couldn't find it in the docs. My k8s/ dir YAML has a reference like this:

image: {{ build.images["Dockerfile"] }}

I'm not sure what the right answer to this problem is, but here's the situation I am in. We have a Dockerized PHP application. We deploy it as pods with two containers: one running php-fpm, the other running nginx with a fastcgi_proxy. As far as I know, this is the best way to put a PHP application into Kubernetes.

An example directory structure:

myrepo.git
├── containers
│   └── proxy
│       ├── Dockerfile
│       └── nginx.conf
├── public
├── src
│       └── ...
└── Dockerfile

And the nginx proxy's Dockerfile:

# ./containers/proxy/Dockerfile
FROM nginx:1.13-alpine

COPY ./nginx.conf /etc/nginx/nginx.conf

RUN mkdir -p /srv/public
COPY ../../public /srv/public

WORKDIR /srv

The problem is that, currently, I have to build that Dockerfile with its context as the repository root (in order to copy in the ./public folder). Forge defaults the build context as ./containers/proxy, where I need it to be ./

Take a look at the output below... I was logged into my Docker daemon as d6eautomaton but told Forge to use my plombardi89 credentials. Forge couldn't authenticate so it isn't using the credentials I provided it.

To make Forge succeed I had to docker login -u plombardi89 ...

(virtualenv) [plombardi@localhost devops-days-boston]$ forge setup
== Checking Kubernetes Setup ==

CMD: kubectl version --short -> Client Version: v1.7.4
  Server Version: v1.7.5
CMD: kubectl get service kubernetes --namespace default -> NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  kubernetes   10.0.0.1     <none>        443/TCP   1d

== Setting up Docker ==

Docker registry[registry.hub.docker.com]: 
Docker user[plombardi]: plombardi89
Docker organization[plombardi89]: 
Docker password: 

pull: registry.hub.docker.com/datawire/forge-setup-test:1
tag: registry.hub.docker.com/datawire/forge-setup-test:1 forge_test dummy
push: forge_test dummy -> registry.hub.docker.com/plombardi89/forge_test:dummy

== Writing config to forge.yaml ==

# Global forge configuration
# DO NOT CHECK INTO GITHUB, THIS FILE CONTAINS SECRETS
workdir: work
docker-repo: registry.hub.docker.com/plombardi89
user: plombardi89
password: >
  REDACTED

== Done ==
(virtualenv) [plombardi@localhost devops-days-boston]$ forge deploy
forge: 2 child task(s) errored
  scan: /home/plombardi/work/devops-days-boston -> hello, quote
  dependencies: hello, quote ->
  service: hello -> 1 child task(s) errored
    build: hello:7401904885f14a552709a54751d4046bfd37b762.ephemeral -> 1 child task(s) errored
      bake: built Dockerfile
        docker-build: /home/plombardi/work/devops-days-boston/hello/ /home/plombardi/work/devops-days-boston/hello/Dockerfile hello 7401904885f14a552709a54751d4046bfd37b762.ephemeral -> 1 child 
task(s) errored
          CMD: command failed[1]: Sending build context to Docker daemon 7.168 kB
            Step 1 : FROM python:3-alpine
            Get https://registry-1.docker.io/v2/library/python/manifests/3-alpine: unauthorized: incorrect username or password
  service: quote -> 1 child task(s) errored
    build: quote:282e17e182706eb6184ea8c3aeaeef0f460815f8.ephemeral -> 1 child task(s) errored
      bake: built Dockerfile
        docker-build: /home/plombardi/work/devops-days-boston/quote/ /home/plombardi/work/devops-days-boston/quote/Dockerfile quote 282e17e182706eb6184ea8c3aeaeef0f460815f8.ephemeral -> 1 child 
task(s) errored
          CMD: command failed[1]: Sending build context to Docker daemon 4.608 kB
            Step 1 : FROM python:3-alpine
            Get https://registry-1.docker.io/v2/library/python/manifests/3-alpine: unauthorized: incorrect username or password
(virtualenv) [plombardi@localhost devops-days-boston]$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (d6eautomaton): plombardi89
Password: 
Login Succeeded
(virtualenv) [plombardi@localhost devops-days-boston]$ forge deploy
scan: /home/plombardi/work/devops-days-boston -> hello, quote
dependencies: hello, quote ->
service: hello
  build: Dockerfile
    pushed registry.hub.docker.com/plombardi89/hello:7401904885f14a552709a54751d4046bfd37b762.ephemeral
    manifests /home/plombardi/work/devops-days-boston/work/k8s/hello
  deploy: kubectl apply -f /home/plombardi/work/devops-days-boston/work/k8s/hello -> OK
    service "hello" created
    deployment "hello-kubernetes" created
service: quote
  build: Dockerfile
    pushed registry.hub.docker.com/plombardi89/quote:282e17e182706eb6184ea8c3aeaeef0f460815f8.ephemeral
    manifests /home/plombardi/work/devops-days-boston/work/k8s/quote
  deploy: kubectl apply -f /home/plombardi/work/devops-days-boston/work/k8s/quote -> OK
    service "quote" created
    deployment "quote" created

The "parallel" output is pretty cool to watch, but it's verbose and hard to follow. When things break it's quite difficult to figure out what broke and why, while text is flying around the screen.

I believe there's a middle ground of informative and actionable that could be found with the CLI output.

Many other tools follow the convention of using dot-file directories for storing local project configuration, caches and outputs. Forge should do the same thing. The directory "work" is pretty common and likely to be used already for something by a project.

Other tools:

• git (.git)
• Gradle (.gradle)
• Intellij (.idea)

I can probably think of others, but the convention stands.

I just committed my Docker Hub password to source control (in forge.yaml). Thankfully it's not a public repository and I can delete the source tree but that's really really bad.

Not sure why Forge believes it's a good idea to default to my computer's login name as the username for forge. That's not likely to be a sensible default at all given that there are a whole heck of a lot of plombardi users in the world. My actual username for Docker Hub is plombardi89.

Also it's not clear how Forge would work with a Container Registry that uses non traditional Username/Password login such as Google Container Registry or ECR.

See issue gh-18 for context.

see logs below

This is what you get when you run forge setup inside a Docker container

`

forge setup
--
  | == Checking Kubernetes Setup ==
  |  
  | CMD: kubectl version --short -> Client Version: v1.6.0
  | Server Version: v1.6.0
  | CMD: kubectl get service kubernetes --namespace default -> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  | kubernetes 10.96.0.1 443/TCP 10m
  | Traceback (most recent call last):
  | File "/home/scrapbook/.bin/forge/bin/forge", line 11, in
  | load_entry_point('Forge==0.2.6+1.g161789d', 'console_scripts', 'forge')()
  | File "/home/scrapbook/.bin/forge/venv/local/lib/python2.7/site-packages/forge/cli.py", line 388, in call_main
  | exit(main())
  | File "/home/scrapbook/.bin/forge/venv/local/lib/python2.7/site-packages/forge/cli.py", line 350, in main
  | if args["setup"]: return forge.setup()
  | File "/home/scrapbook/.bin/forge/venv/local/lib/python2.7/site-packages/forge/cli.py", line 124, in setup
  | user = os.environ["USER"]
  | File "/home/scrapbook/.bin/forge/venv/lib/python2.7/UserDict.py", line 23, in getitem
  | raise KeyError(key)
  | KeyError: 'USER'

`

When running my application deploys, a lot of how they are configured depends on the target environment. For example, running MySQL locally I might only want to allocate 1GB of memory, but in production it would be much higher.

One of the ways I was thinking this could be possible, is if the service.yml allowed Jinja2 templating in the configuration values. This could be a powerful feature, especially if it were combined with a Jinja2 filter that allowed you to pull an environment variable. Imagine this:

# ./service.yml
name: my-microservice
memory: "{{ env('MEMORY') | default('0.5G') }}"

This could of course be handled in the K8S template, too:

# ./k8s/deployment.yaml
# ...
         resources:
           limits:
             memory: {{ env('MEMORY') | default('0.5G') }}
             cpu: {{ env('CPU') | default('0.25') }}
# ...

But I think the self documenting nature of the first example, by being contained in service.yml, is more advantageous.

This would allow you to use a local .env file in development, with something like source .env && forge deploy. It would also be helpful for handling deploys in CI environments, without storing production values in the Git repository.

First up, kudos to the entire team as forge is awesome !

Let's say the container registry is this :- gitlab.example.com:4567/username/userproject

In this case, the forget setup would fail even after the image push/pull has succeeded because the validation in docker.py:registry_get gets a 404 as it's looking at an incorrect URL

The URL that gets constructed in docker.py:registry_get is gitlab.example.com:4567/username/userproject/v2/username/forge_test/manifests/dummy which is incorrect and the API returns a 404 which doesn't get handled correctly.

It should rather be gitlab.example.com:4567/v2/username/userproject/username/forge_test/manifests/dummy

[plombardi@localhost devops-days-boston]$ forge setup
== Checking Kubernetes Setup ==

CMD: command failed[1]: Client Version: v1.7.4
  The connection to the server localhost:8080 was refused - did you specify the right host or port?

== Kubernetes Check Failed ==

Please make sure kubectl is installed/configured correctly.
[plombardi@localhost devops-days-boston]$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

I think Forge is being a little too overzealous about sanity checking. It's a perfectly valid state to be in where I have kubectl installed but I do not have an active cluster running (e.g. Minikube not started, previous Kubernaut cluster is expired).

If a service.yaml is in an existing directory, even if it's not a forge service.yaml, Forge tries to parse it and it barfs.

Istio only supports ClusterIP services.

Any HTTP service in Istio has to have a name on its service port, and that name has to start with http- Or just be http.

Simple test case is to use Forge with the BookInfo app.

Hello,
Just started with forge quickstart page but script return:

-->  Checking for python: OK
-->  Checking for virtualenv: FAIL

        Cannot find virtualenv, please install and try again.

Documentation should mention that virtualenv is needed

pip install virtualenv

did the trick:

--> Performing installation environment sanity checks...
-->  Checking for python: OK
-->  Checking for virtualenv: OK
-->  Checking install target: OK
--> Creating /Users/fmedery/forge installation directory...
--> Installing Forge...
--> Installed!

  Forge has been installed into '/Users/fmedery/forge'. You may want to
  add '/Users/fmedery/forge/bin' to your PATH.

This would be convenient if you make a change in a subdirectory of your service.

The command forge setup is confusing. Every Docker Registry implementation in existence uses the following format for a repository name: $namespace/$image_name. When running forge setup you're prompted for Docker repo: and the inclination is to put in something like plombardi89/helloworld, however, if you do that then the forge setup fails later on:

Command[ERR[1]]: docker push registry.hub.docker.com/plombardi89/bookinfo/forge_test:dummy
  The push refers to a repository [registry.hub.docker.com/plombardi89/bookinfo/forge_test]
  3fb66f713c9f: Preparing
  error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!doctype html>\n<html class=\"no-js\" lang=\"\">\n    <head>\n        <meta

The problem is Forge is expecting just the $namespace part of the image name (e.g. plombardi89) and not the the $image_name bit which it will create for the project based on the current working directories name.

It would be nice to have VCS (Git) Branch Awareness built into Forge so that for example if several development feature branches are open for a service they can all be deployed at once.

I have been doing a lot of thinking around the use of namespaces recently. It seems ideally the proper use of a namespace is to make them at a minimum application centric so that it's not possible for different applications or teams to conflict on a common name inside of a namespace (e.g. consider what happens if two teams or applications want to have something named api).

So that leaves us with at a minimum most folks should use something like below:

namespace: "${APP_NAME}"  # "kubernaut"

If you're like us and want to run multiple operational environments out of the same cluster then you also likely need to append an environment name to the namespace, for example, we use a single cluster for prod and staging.

namespace: "${APP_NAME}-${APP_ENV}"  # "kubernaut-prod" or "kubernaut-rc"

Now consider you have a situation where two or three developers hack on a single Application composed of microservices all under the same repository. They use branch names like dev/cool-new-feature1. It would be nice if Forge could have a way to just generate the proper namespace to deploy into. Some tools do something like ${APP_NAME}-${Eight_Character_String} to ensure uniqueness but other ideas would be to do ${APP_NAME}-${BRANCH_NAME}`

Not sure if there is already an option to hand-craft the docker image build name with chosen metadata (example: deployment 'track', version etc) in which case this is probably not needed.

Else it'd be good to differentiate between different images to help perform certain administrative actions.

When searching for a dependency, Forge tries looking up repositories using the current repo's origin. Sometimes, the repository might not be under the same organization, or even have the same name. It would be helpful to somehow be able to control what Forge searches for when looking up a particular dependency.

Forge currently just uses the entire source tree but it really should respect the gitignore otherwise the hash will never be the same as the actual git hash for the source tree.

Not sure if there is a python implementation of the gitignore rules, but another option is to use the git command itself which offers powerful capabilities for inspecting ignored files:

https://stackoverflow.com/questions/466764/git-command-to-show-which-specific-files-are-ignored-by-gitignore

Forge can output Helm charts, and not just the deployment manifests

Jinja2 should error when it sees a variable in a template that it does not have defined. This would prevent empty interpolations from occurring.

Summary

Microservices are often dependent on other microservices. If A -> B -> C and someone uses Forge to start A then B and C should be started as well.

Proposal

Add metadata to service.yaml that allows encoding of service dependencies:

requires:
  - service1
  - service2

Then make forge deploy smart enough to expand the set of services being operated on to include dependencies. Because this is a network of services, this should be smart enough to handle dependency cycles. (These are organically grown runtime dependencies, not build time dependencies, so cycles are expected/ok.)

Add a concept of a workspace to forge. This can start out as whatever directory the forge.yaml is in. When forge is searching for services, it will first check the workspace, and then check git in order to deploy any dependent services.

The combination of these features allows forge to function as a "workspace manager" (kinda like an IDE in some ways) for microservices development.

Rationale

Based on at least one reddit post, people wonder about how to do this, so it is useful functionality and plausibly contributes to conversion. This is relatively cheap to implement and makes for a good blog aritcle and so contributes towards attention. This could be the basis of a combined example project that features ambassador + a network of services with a CI/CD pipeline that leverages both forge and ambassador, and so it contributes towards "Better Together".

Historical Context from Private Repository Comments

plombardi

How do you plan to actually handle dependency cycles? I'm not sure it's realistically doable or if you do attempt to solve that problem I think we might want to consider having a dumb option that ignores cycle detection and lets the services and Kubernetes use it's default strategy of start and fail until dependencies come up.

rhs

Mostly I just mean make sure forge doesn't stack overflow if there are dependency cycles. This would be necessary regardless of whether they are legal or not.

My thinking about cycles is this. Let's say there are two classes of dependencies: startup (service A needs service B to be operational in order to start) and runtime (service A doesn't need service B to start, but it does need service B to serve requests in some situations).

Obviously for startup dependencies, cycles are not possible to handle and should be illegal. For runtime dependencies, cycles aren't an issue since you just start everything up in any order and make sure it's all running before you hit any given service.

My comments were assuming that what we would start with was runtime dependencies and add in startup dependencies later.

Right now Forge always prompts for username/password for a Docker registry. There is a better way and it would allow us to remove the base64 encoded password from forge.yaml.

Docker logins are stored in ~/.docker/config.json and are keyed by the registry hostname. If we are given a known registry that is in the Docker config we can just use those credentials. If the registry and credentials are not present then we can execute docker login ... and ensure the credentials are present in the future.

I run forge deploy of the todo app against an empty cluster. Not all of my services deploy. I repeat forge deploy. More services deploy. I run it again. Finally, all my services deploy.

The error messages I get from the first run are below (I get them in the second run as well, but not in the final run).

MacBook-Pro-23:todo richard$ forge deploy
forge: 3 child task(s) errored
  scan: /Users/richard/dw/todo -> api, auth, todo-db, prometheus, search, tasks
  dependencies: api, auth, todo-db, prometheus, search, tasks ->
  service: api -> 1 child task(s) errored
    build: api:052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: api 052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
  service: auth -> 1 child task(s) errored
    build: auth:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: auth 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
  service: todo-db
    build: manifests /Users/richard/dw/todo/work/k8s/todo-db
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/todo-db -> OK
      storageclass "fast" created
      service "todo-db" created
      statefulset "todo-db" created
  service: prometheus
    build: pushed registry.hub.docker.com/richarddli/prometheus:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git
      manifests /Users/richard/dw/todo/work/k8s/prometheus
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/prometheus -> OK
      service "prometheus" created
      deployment "prometheus" created
      configmap "prometheus-config" created
  service: search
    build: manifests /Users/richard/dw/todo/work/k8s/search
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/search -> OK
      service "search" created
      deployment "search" created
  service: tasks -> 1 child task(s) errored
    build: tasks:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: tasks 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`

forge --help doesn't give much information about what bake, push, etc do. Would help to have more detailed documentation on the commands and their arguments.

Reading #42 reminded me of a pattern I was thinking about implementing with my Dockerfiles with regard to multi-stage builds. These two might be slightly related, if the build target could be placed into the image tag.

I was thinking about adding a build stage that includes the project debug/development dependencies preinstalled. Something like:

FROM php:7.1 AS debug

# ... make the debug build

FROM php:7.1 AS release

COPY --from=debug /app /app
# ... prune out debug stuff

If we could configure the build target, then I could do something like:

containers:
  - dockerfile: ./Dockerfile
    target: {{ 'debug' if env.DEBUG else 'release' }}

and then with my proposed solution in #42, you could do this to avoid collisions on images:

containers:
  - dockerfile: ./Dockerfile
    tag: {{version}}{%if env.DEBUG %}-debug{% endif %}
    target: {{ 'debug' if env.DEBUG else 'release' }}

Not sure this is the best solution, but if you are depending on forge to pull repositories for other services, it would be nice to have an easy way to pull the latest versions of all the requirements.

name: root-level
requires:
  - api
  - backend
  - worker

forge pull would pull the latest versions of api, backend and worker from Git, as well as any of their dependencies

There might be a reason to have forge pull [service], to control which service gets updated, but maybe not initially

The forge build command which converts source to Docker images is painfully slow for projects that need to pull dependencies and compile from source for changes. There is no easy or good way to cache the dependencies or do an incremental build which means that the speed benefit of Forge is lost immediately for these types of languages.

The Docker build itself took 46.36 seconds for a very simple "Hello, world!" type application.
Forge itself took 57.16 seconds.

If I wasn't running the build of source code in a Dockerfile and could reuse my local build caches then my build time drops massively. Here is a Gradle run:

plombardi@palwork ~/w/kubernaut-monorepo> ./gradlew clean test shadowJar

> Task :common:compileKotlin
Using kotlin incremental compilation

> Task :poolman:compileKotlin
Using kotlin incremental compilation

> Task :web:compileKotlin
Using kotlin incremental compilation


BUILD SUCCESSFUL in 3s
13 actionable tasks: 13 executed

It is very easy to accidentally commit forge.yaml with your password; Forge should probably detect if it is in a Git repository and either generate or modify the .gitignore in the same directory as forge.yaml if necessary.

Probably want to e.g. check if /var/run/docker.sock is readable/writeable and then prefix sudo if it isn't:

https://github.com/datawire/telepresence/blob/38b808dff4ed45fdb0635eaa2418d3020169da1a/cli/telepresence#L152

I'm gonna start off by saying that I'm not sure if there's a problem or a solution here. I guess I just wanted to have a discussion about the semantics of templating with Forge. Right now, templates are considered Jinja2 first, and then YAML. When I asked about this on Gitter, @rhs mentioned he wasn't sure if this was the right choice, it just was how he had implemented it at first. I think it might work out to be the best implementation.

• What are the advantages of each approach? (YAML file with Jinja2 values, or a Jinja2 template rendering a YAML file)
• What existing examples of templated YAML are there?
  • Ansible (YAML file with Jinja2 values)
  • ???

I originally typed out a long example situation I ran into but it ended up not being relevant, so I boiled this issue down into some discussion points that might be good to have.

Namespaces need to exist before anything else can use them which means they need to be created before any other resources are created if those resources explicitly depend on the namespace

This is not that important, kind of cosmetic. I like to put all my "sub" container images in ./containers folder, e.g. ./containers/proxy. For a project named my-api, this creates an image named my-api-containers-proxy. I'd like to just name it my-api-proxy.

containers:
  - name: my-api-proxy
    dockerfile: containers/proxy/Dockerfile