datawire / forge Goto Github PK
View Code? Open in Web Editor NEWDefine and run multi-container apps in Kubernetes
Home Page: http://forge.sh
License: Apache License 2.0
Define and run multi-container apps in Kubernetes
Home Page: http://forge.sh
License: Apache License 2.0
Would be nice to have this in Datawire's Homebrew tap for easy installation on macOS
It would be really nice to be able to add custom code before and after commands. A real world use case of this would be to do something like istioctl kube-inject
after forge build ...
or forge deploy ...
. However, other use cases would be to glue in custom code that goes off and does something like send a Slack message or make an API call to another service.
Spit balling a bit of implementation I would like to see this work similar to how cookiecutter
handles hooks: There is a directory called hooks
which can have an arbitrary executable programs dropped into it. Some consideration will need to make, however, to account for the different commands in forge
and also that there is likely some amount of ordering that needs to be followed.
It's not clear what I am supposed to do here... and I couldn't find it in the docs. My k8s/
dir YAML has a reference like this:
image: {{ build.images["Dockerfile"] }}
I'm not sure what the right answer to this problem is, but here's the situation I am in. We have a Dockerized PHP application. We deploy it as pods with two containers: one running php-fpm, the other running nginx with a fastcgi_proxy. As far as I know, this is the best way to put a PHP application into Kubernetes.
An example directory structure:
myrepo.git
├── containers
│ └── proxy
│ ├── Dockerfile
│ └── nginx.conf
├── public
├── src
│ └── ...
└── Dockerfile
And the nginx proxy's Dockerfile:
# ./containers/proxy/Dockerfile
FROM nginx:1.13-alpine
COPY ./nginx.conf /etc/nginx/nginx.conf
RUN mkdir -p /srv/public
COPY ../../public /srv/public
WORKDIR /srv
The problem is that, currently, I have to build that Dockerfile with its context as the repository root (in order to copy in the ./public
folder). Forge defaults the build context as ./containers/proxy
, where I need it to be ./
Take a look at the output below... I was logged into my Docker daemon as d6eautomaton
but told Forge to use my plombardi89
credentials. Forge couldn't authenticate so it isn't using the credentials I provided it.
To make Forge succeed I had to docker login -u plombardi89 ...
(virtualenv) [plombardi@localhost devops-days-boston]$ forge setup
== Checking Kubernetes Setup ==
CMD: kubectl version --short -> Client Version: v1.7.4
Server Version: v1.7.5
CMD: kubectl get service kubernetes --namespace default -> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1 <none> 443/TCP 1d
== Setting up Docker ==
Docker registry[registry.hub.docker.com]:
Docker user[plombardi]: plombardi89
Docker organization[plombardi89]:
Docker password:
pull: registry.hub.docker.com/datawire/forge-setup-test:1
tag: registry.hub.docker.com/datawire/forge-setup-test:1 forge_test dummy
push: forge_test dummy -> registry.hub.docker.com/plombardi89/forge_test:dummy
== Writing config to forge.yaml ==
# Global forge configuration
# DO NOT CHECK INTO GITHUB, THIS FILE CONTAINS SECRETS
workdir: work
docker-repo: registry.hub.docker.com/plombardi89
user: plombardi89
password: >
REDACTED
== Done ==
(virtualenv) [plombardi@localhost devops-days-boston]$ forge deploy
forge: 2 child task(s) errored
scan: /home/plombardi/work/devops-days-boston -> hello, quote
dependencies: hello, quote ->
service: hello -> 1 child task(s) errored
build: hello:7401904885f14a552709a54751d4046bfd37b762.ephemeral -> 1 child task(s) errored
bake: built Dockerfile
docker-build: /home/plombardi/work/devops-days-boston/hello/ /home/plombardi/work/devops-days-boston/hello/Dockerfile hello 7401904885f14a552709a54751d4046bfd37b762.ephemeral -> 1 child
task(s) errored
CMD: command failed[1]: Sending build context to Docker daemon 7.168 kB
Step 1 : FROM python:3-alpine
Get https://registry-1.docker.io/v2/library/python/manifests/3-alpine: unauthorized: incorrect username or password
service: quote -> 1 child task(s) errored
build: quote:282e17e182706eb6184ea8c3aeaeef0f460815f8.ephemeral -> 1 child task(s) errored
bake: built Dockerfile
docker-build: /home/plombardi/work/devops-days-boston/quote/ /home/plombardi/work/devops-days-boston/quote/Dockerfile quote 282e17e182706eb6184ea8c3aeaeef0f460815f8.ephemeral -> 1 child
task(s) errored
CMD: command failed[1]: Sending build context to Docker daemon 4.608 kB
Step 1 : FROM python:3-alpine
Get https://registry-1.docker.io/v2/library/python/manifests/3-alpine: unauthorized: incorrect username or password
(virtualenv) [plombardi@localhost devops-days-boston]$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (d6eautomaton): plombardi89
Password:
Login Succeeded
(virtualenv) [plombardi@localhost devops-days-boston]$ forge deploy
scan: /home/plombardi/work/devops-days-boston -> hello, quote
dependencies: hello, quote ->
service: hello
build: Dockerfile
pushed registry.hub.docker.com/plombardi89/hello:7401904885f14a552709a54751d4046bfd37b762.ephemeral
manifests /home/plombardi/work/devops-days-boston/work/k8s/hello
deploy: kubectl apply -f /home/plombardi/work/devops-days-boston/work/k8s/hello -> OK
service "hello" created
deployment "hello-kubernetes" created
service: quote
build: Dockerfile
pushed registry.hub.docker.com/plombardi89/quote:282e17e182706eb6184ea8c3aeaeef0f460815f8.ephemeral
manifests /home/plombardi/work/devops-days-boston/work/k8s/quote
deploy: kubectl apply -f /home/plombardi/work/devops-days-boston/work/k8s/quote -> OK
service "quote" created
deployment "quote" created
The "parallel" output is pretty cool to watch, but it's verbose and hard to follow. When things break it's quite difficult to figure out what broke and why, while text is flying around the screen.
I believe there's a middle ground of informative and actionable that could be found with the CLI output.
Many other tools follow the convention of using dot-file directories for storing local project configuration, caches and outputs. Forge should do the same thing. The directory "work" is pretty common and likely to be used already for something by a project.
Other tools:
.git
).gradle
).idea
)I can probably think of others, but the convention stands.
I just committed my Docker Hub password to source control (in forge.yaml
). Thankfully it's not a public repository and I can delete the source tree but that's really really bad.
Not sure why Forge believes it's a good idea to default to my computer's login name as the username for forge. That's not likely to be a sensible default at all given that there are a whole heck of a lot of plombardi
users in the world. My actual username for Docker Hub is plombardi89
.
Also it's not clear how Forge would work with a Container Registry that uses non traditional Username/Password login such as Google Container Registry or ECR.
See issue gh-18 for context.
see logs below
This is what you get when you run forge setup inside a Docker container
`
forge setup
--
| == Checking Kubernetes Setup ==
|
| CMD: kubectl version --short -> Client Version: v1.6.0
| Server Version: v1.6.0
| CMD: kubectl get service kubernetes --namespace default -> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
| kubernetes 10.96.0.1 443/TCP 10m
| Traceback (most recent call last):
| File "/home/scrapbook/.bin/forge/bin/forge", line 11, in
| load_entry_point('Forge==0.2.6+1.g161789d', 'console_scripts', 'forge')()
| File "/home/scrapbook/.bin/forge/venv/local/lib/python2.7/site-packages/forge/cli.py", line 388, in call_main
| exit(main())
| File "/home/scrapbook/.bin/forge/venv/local/lib/python2.7/site-packages/forge/cli.py", line 350, in main
| if args["setup"]: return forge.setup()
| File "/home/scrapbook/.bin/forge/venv/local/lib/python2.7/site-packages/forge/cli.py", line 124, in setup
| user = os.environ["USER"]
| File "/home/scrapbook/.bin/forge/venv/lib/python2.7/UserDict.py", line 23, in getitem
| raise KeyError(key)
| KeyError: 'USER'
`
When running my application deploys, a lot of how they are configured depends on the target environment. For example, running MySQL locally I might only want to allocate 1GB of memory, but in production it would be much higher.
One of the ways I was thinking this could be possible, is if the service.yml
allowed Jinja2 templating in the configuration values. This could be a powerful feature, especially if it were combined with a Jinja2 filter that allowed you to pull an environment variable. Imagine this:
# ./service.yml
name: my-microservice
memory: "{{ env('MEMORY') | default('0.5G') }}"
This could of course be handled in the K8S template, too:
# ./k8s/deployment.yaml
# ...
resources:
limits:
memory: {{ env('MEMORY') | default('0.5G') }}
cpu: {{ env('CPU') | default('0.25') }}
# ...
But I think the self documenting nature of the first example, by being contained in service.yml, is more advantageous.
This would allow you to use a local .env
file in development, with something like source .env && forge deploy
. It would also be helpful for handling deploys in CI environments, without storing production values in the Git repository.
First up, kudos to the entire team as forge
is awesome !
Let's say the container registry is this :- gitlab.example.com:4567/username/userproject
In this case, the forget setup would fail even after the image push/pull has succeeded because the validation in docker.py:registry_get
gets a 404 as it's looking at an incorrect URL
The URL that gets constructed in docker.py:registry_get
is gitlab.example.com:4567/username/userproject/v2/username/forge_test/manifests/dummy
which is incorrect and the API returns a 404 which doesn't get handled correctly.
It should rather be gitlab.example.com:4567/v2/username/userproject/username/forge_test/manifests/dummy
[plombardi@localhost devops-days-boston]$ forge setup
== Checking Kubernetes Setup ==
CMD: command failed[1]: Client Version: v1.7.4
The connection to the server localhost:8080 was refused - did you specify the right host or port?
== Kubernetes Check Failed ==
Please make sure kubectl is installed/configured correctly.
[plombardi@localhost devops-days-boston]$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
I think Forge is being a little too overzealous about sanity checking. It's a perfectly valid state to be in where I have kubectl installed but I do not have an active cluster running (e.g. Minikube not started, previous Kubernaut cluster is expired).
If a service.yaml
is in an existing directory, even if it's not a forge service.yaml
, Forge tries to parse it and it barfs.
Istio only supports ClusterIP
services.
Any HTTP service in Istio has to have a name
on its service port
, and that name has to start with http-
Or just be http
.
Simple test case is to use Forge with the BookInfo app.
Hello,
Just started with forge quickstart page but script return:
--> Checking for python: OK
--> Checking for virtualenv: FAIL
Cannot find virtualenv, please install and try again.
Documentation should mention that virtualenv is needed
pip install virtualenv
did the trick:
--> Performing installation environment sanity checks...
--> Checking for python: OK
--> Checking for virtualenv: OK
--> Checking install target: OK
--> Creating /Users/fmedery/forge installation directory...
--> Installing Forge...
--> Installed!
Forge has been installed into '/Users/fmedery/forge'. You may want to
add '/Users/fmedery/forge/bin' to your PATH.
This would be convenient if you make a change in a subdirectory of your service.
The command forge setup
is confusing. Every Docker Registry implementation in existence uses the following format for a repository name: $namespace/$image_name
. When running forge setup
you're prompted for Docker repo:
and the inclination is to put in something like plombardi89/helloworld
, however, if you do that then the forge setup
fails later on:
Command[ERR[1]]: docker push registry.hub.docker.com/plombardi89/bookinfo/forge_test:dummy
The push refers to a repository [registry.hub.docker.com/plombardi89/bookinfo/forge_test]
3fb66f713c9f: Preparing
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!doctype html>\n<html class=\"no-js\" lang=\"\">\n <head>\n <meta
The problem is Forge is expecting just the $namespace
part of the image name (e.g. plombardi89
) and not the the $image_name
bit which it will create for the project based on the current working directories name.
It would be nice to have VCS (Git) Branch Awareness built into Forge so that for example if several development feature branches are open for a service they can all be deployed at once.
I have been doing a lot of thinking around the use of namespaces recently. It seems ideally the proper use of a namespace is to make them at a minimum application centric so that it's not possible for different applications or teams to conflict on a common name inside of a namespace (e.g. consider what happens if two teams or applications want to have something named api
).
So that leaves us with at a minimum most folks should use something like below:
namespace: "${APP_NAME}" # "kubernaut"
If you're like us and want to run multiple operational environments out of the same cluster then you also likely need to append an environment name to the namespace, for example, we use a single cluster for prod
and staging
.
namespace: "${APP_NAME}-${APP_ENV}" # "kubernaut-prod" or "kubernaut-rc"
Now consider you have a situation where two or three developers hack on a single Application composed of microservices all under the same repository. They use branch names like dev/cool-new-feature1
. It would be nice if Forge could have a way to just generate the proper namespace to deploy into. Some tools do something like ${APP_NAME}-${Eight_Character_String} to ensure uniqueness but other ideas would be to do
${APP_NAME}-${BRANCH_NAME}`
Not sure if there is already an option to hand-craft the docker image build name with chosen metadata (example: deployment 'track', version etc) in which case this is probably not needed.
Else it'd be good to differentiate between different images to help perform certain administrative actions.
When searching for a dependency, Forge tries looking up repositories using the current repo's origin. Sometimes, the repository might not be under the same organization, or even have the same name. It would be helpful to somehow be able to control what Forge searches for when looking up a particular dependency.
Forge currently just uses the entire source tree but it really should respect the gitignore otherwise the hash will never be the same as the actual git hash for the source tree.
Not sure if there is a python implementation of the gitignore rules, but another option is to use the git
command itself which offers powerful capabilities for inspecting ignored files:
Forge can output Helm charts, and not just the deployment manifests
Jinja2 should error when it sees a variable in a template that it does not have defined. This would prevent empty interpolations from occurring.
Microservices are often dependent on other microservices. If A -> B -> C
and someone uses Forge to start A
then B
and C
should be started as well.
Add metadata to service.yaml
that allows encoding of service dependencies:
requires:
- service1
- service2
Then make forge deploy smart enough to expand the set of services being operated on to include dependencies. Because this is a network of services, this should be smart enough to handle dependency cycles. (These are organically grown runtime dependencies, not build time dependencies, so cycles are expected/ok.)
Add a concept of a workspace to forge. This can start out as whatever directory the forge.yaml is in. When forge is searching for services, it will first check the workspace, and then check git in order to deploy any dependent services.
The combination of these features allows forge to function as a "workspace manager" (kinda like an IDE in some ways) for microservices development.
Based on at least one reddit post, people wonder about how to do this, so it is useful functionality and plausibly contributes to conversion. This is relatively cheap to implement and makes for a good blog aritcle and so contributes towards attention. This could be the basis of a combined example project that features ambassador + a network of services with a CI/CD pipeline that leverages both forge and ambassador, and so it contributes towards "Better Together".
plombardi
How do you plan to actually handle dependency cycles? I'm not sure it's realistically doable or if you do attempt to solve that problem I think we might want to consider having a dumb option that ignores cycle detection and lets the services and Kubernetes use it's default strategy of start and fail until dependencies come up.
rhs
Mostly I just mean make sure forge doesn't stack overflow if there are dependency cycles. This would be necessary regardless of whether they are legal or not.
My thinking about cycles is this. Let's say there are two classes of dependencies: startup (service A needs service B to be operational in order to start) and runtime (service A doesn't need service B to start, but it does need service B to serve requests in some situations).
Obviously for startup dependencies, cycles are not possible to handle and should be illegal. For runtime dependencies, cycles aren't an issue since you just start everything up in any order and make sure it's all running before you hit any given service.
My comments were assuming that what we would start with was runtime dependencies and add in startup dependencies later.
Right now Forge always prompts for username/password for a Docker registry. There is a better way and it would allow us to remove the base64 encoded password from forge.yaml
.
Docker logins are stored in ~/.docker/config.json
and are keyed by the registry hostname. If we are given a known registry that is in the Docker config we can just use those credentials. If the registry and credentials are not present then we can execute docker login ...
and ensure the credentials are present in the future.
I run forge deploy of the todo app against an empty cluster. Not all of my services deploy. I repeat forge deploy. More services deploy. I run it again. Finally, all my services deploy.
The error messages I get from the first run are below (I get them in the second run as well, but not in the final run).
MacBook-Pro-23:todo richard$ forge deploy
forge: 3 child task(s) errored
scan: /Users/richard/dw/todo -> api, auth, todo-db, prometheus, search, tasks
dependencies: api, auth, todo-db, prometheus, search, tasks ->
service: api -> 1 child task(s) errored
build: api:052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
push: 1 child task(s) errored
push: api 052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
service: auth -> 1 child task(s) errored
build: auth:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
push: 1 child task(s) errored
push: auth 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
service: todo-db
build: manifests /Users/richard/dw/todo/work/k8s/todo-db
deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/todo-db -> OK
storageclass "fast" created
service "todo-db" created
statefulset "todo-db" created
service: prometheus
build: pushed registry.hub.docker.com/richarddli/prometheus:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git
manifests /Users/richard/dw/todo/work/k8s/prometheus
deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/prometheus -> OK
service "prometheus" created
deployment "prometheus" created
configmap "prometheus-config" created
service: search
build: manifests /Users/richard/dw/todo/work/k8s/search
deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/search -> OK
service "search" created
deployment "search" created
service: tasks -> 1 child task(s) errored
build: tasks:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
push: 1 child task(s) errored
push: tasks 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
forge --help
doesn't give much information about what bake
, push
, etc do. Would help to have more detailed documentation on the commands and their arguments.
Reading #42 reminded me of a pattern I was thinking about implementing with my Dockerfiles with regard to multi-stage builds. These two might be slightly related, if the build target could be placed into the image tag.
I was thinking about adding a build stage that includes the project debug/development dependencies preinstalled. Something like:
FROM php:7.1 AS debug
# ... make the debug build
FROM php:7.1 AS release
COPY --from=debug /app /app
# ... prune out debug stuff
If we could configure the build target, then I could do something like:
containers:
- dockerfile: ./Dockerfile
target: {{ 'debug' if env.DEBUG else 'release' }}
and then with my proposed solution in #42, you could do this to avoid collisions on images:
containers:
- dockerfile: ./Dockerfile
tag: {{version}}{%if env.DEBUG %}-debug{% endif %}
target: {{ 'debug' if env.DEBUG else 'release' }}
Not sure this is the best solution, but if you are depending on forge to pull repositories for other services, it would be nice to have an easy way to pull the latest versions of all the requirements.
name: root-level
requires:
- api
- backend
- worker
forge pull
would pull the latest versions of api
, backend
and worker
from Git, as well as any of their dependencies
There might be a reason to have forge pull [service]
, to control which service gets updated, but maybe not initially
The forge build
command which converts source to Docker images is painfully slow for projects that need to pull dependencies and compile from source for changes. There is no easy or good way to cache the dependencies or do an incremental build which means that the speed benefit of Forge is lost immediately for these types of languages.
The Docker build itself took 46.36
seconds for a very simple "Hello, world!" type application.
Forge itself took 57.16
seconds.
If I wasn't running the build of source code in a Dockerfile and could reuse my local build caches then my build time drops massively. Here is a Gradle run:
plombardi@palwork ~/w/kubernaut-monorepo> ./gradlew clean test shadowJar
> Task :common:compileKotlin
Using kotlin incremental compilation
> Task :poolman:compileKotlin
Using kotlin incremental compilation
> Task :web:compileKotlin
Using kotlin incremental compilation
BUILD SUCCESSFUL in 3s
13 actionable tasks: 13 executed
It is very easy to accidentally commit forge.yaml
with your password; Forge should probably detect if it is in a Git repository and either generate or modify the .gitignore
in the same directory as forge.yaml
if necessary.
Probably want to e.g. check if /var/run/docker.sock
is readable/writeable and then prefix sudo if it isn't:
I'm gonna start off by saying that I'm not sure if there's a problem or a solution here. I guess I just wanted to have a discussion about the semantics of templating with Forge. Right now, templates are considered Jinja2 first, and then YAML. When I asked about this on Gitter, @rhs mentioned he wasn't sure if this was the right choice, it just was how he had implemented it at first. I think it might work out to be the best implementation.
I originally typed out a long example situation I ran into but it ended up not being relevant, so I boiled this issue down into some discussion points that might be good to have.
Namespaces need to exist before anything else can use them which means they need to be created before any other resources are created if those resources explicitly depend on the namespace
This is not that important, kind of cosmetic. I like to put all my "sub" container images in ./containers
folder, e.g. ./containers/proxy
. For a project named my-api
, this creates an image named my-api-containers-proxy
. I'd like to just name it my-api-proxy
.
containers:
- name: my-api-proxy
dockerfile: containers/proxy/Dockerfile
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.