keel-hq / keel Goto Github PK

View Code? Open in Web Editor NEW

2.3K 34.0 279.0 34.97 MB

Kubernetes Operator to automate Helm, DaemonSet, StatefulSet & Deployment updates

Home Page: https://keel.sh

License: Mozilla Public License 2.0

Makefile 0.32% Go 74.67% Shell 0.44% Dockerfile 0.05% JavaScript 8.46% Vue 14.15% Less 1.51% Mustache 0.14% HTML 0.26%

kubernetes docker automate deployment webhook trigger golang keel semver helm

keel's People

Contributors

Stargazers

Watchers

Forkers

prateekpandey14 theckman stickycode 40a htplbc getcloudnative costinm larte azhi satyamz alexandersperling alexxnica kryndex wellcast lendoab wilddima mediataivas alitari gordon-kensci jacobweinstock johnwalker247 cparedes hexsprite azr antiarchitect sanju2312 alonisser zgfh codebreach djsnowsill so0k dln mgk spacemansteve plumdog hk01-digital etsangsplk gillesdemey archifleks leosunmo hassanawanalvi schubter artmello micw cuongtavan faghani yosbeleg89 roarmstrong gfeun kautsig orymate thomaspeitz uzxmx awesomegolang fridayy anorak crazyrex jegan-m cbusbey wlmgithub ashekaramizgm aracki dyhpoon epasham sherrycp phusdu skyionblue boxrick jjlakis karlmutch amq m4rcu5 diegocn mavimo alwinius gilgamezh ggrocco urkonn chrishiestand sirflano qsfang bsord rafaelgaspar andrewlubenets goelprateek auuppp chandanmishra-03 spaghettifunk cguertin14 4k1r4 pgitab ajesse11x wilmardo beracah-group teng1 banthaherder astha-sharma schooloffreelancing chkhd bokysan

keel's Issues

Feature request: deploying Kubernetes StatefulSets

In Kubernetes 1.7, StatefulSets support rolling updates just like Deployments. It would be great if we could use keel to update StatefulSets as well.

transfer keel repo to https://github.com/keel-hq org

Planning to do this with 0.5.0 release (after rc is properly tested)

Support for Slack incoming-webhooks

Good evening,
Do you envision supporting the incoming-webhooks api for slack? It would void having to generate a legacy token with a wide permission scope.
Thanks!

[feature] support for quay.io webhooks

[Poll] Does Keel need a new website?

Hi everyone,

As Keel progressed with features and configuration, our current website https://keel.sh is probably not the easiest place to navigate around.

I am considering to rebuild it based on my other project's website available here: https://webhookrelay.com/. Left side menu should help split installation and configuration guides into different sections.

Unversioned deployment with multiple pods does not update all pods

I have a deployment with 5 replicas following a :latest tag. From the logs, I can see that keel resets the image to 0.0.0 and after 5 seconds applies :latest.

The deployment seems to revert back to the previous version of replicaset, and the rollout does not continue for pods. The single pod that was recreated has the same rc version, but the latest image was pulled via imagePullPolicy. During the reset, I can see 2 pods in ErrImagePull state.

I'm running keel 0.5.0-rc.1 with native webhooks with keel.sh/policy: force in kubernetes 1.7.x

The events from the deployment are:

Normal  ScalingReplicaSet  2m     deployment-controller  Scaled up replica set app-1687492293 to 1
Normal  ScalingReplicaSet  2m     deployment-controller  Scaled down replica set app-3177265212 to 4
Normal  ScalingReplicaSet  2m     deployment-controller  Scaled up replica set app-1687492293 to 2
Normal  ScalingReplicaSet  2m     deployment-controller  Scaled up replica set app-3177265212 to 5
Normal  ScalingReplicaSet  2m     deployment-controller  Scaled down replica set app-1687492293 to 0

and in the end, the pods are:

Name                        RC                             AGE
app-3177265212-40zzc    app-3177265212   2017-10-27T13:26:34Z
app-3177265212-85jsp    app-3177265212   2017-10-27T13:43:07Z   <- This pod has changed
app-3177265212-b2mws    app-3177265212   2017-10-27T13:26:34Z
app-3177265212-p9mkc    app-3177265212   2017-10-27T13:26:34Z
app-3177265212-qth3s    app-3177265212   2017-10-27T13:26:34Z

untagged images should not be ignored

If a deployment file image has no tag specified - keel should not ignore it. Keel should assume that the version from event is newer, therefore deployment files would never be out of date.

spec:
      containers:                    
        - image: karolisr/keel

The downside, however, is that a latest image is required to be available for the repo.

CrashLoopBackOff

I'm not sure if this is a symptom of the other two bugs I've logged, or an independent issue. Feel free to close if it is a duplicate.

Ignoring the repository version and assuming SemVer means 10.4 -> 10.4.0

I use SemVer 'almost', but unless there is actual need for a patch version I omit it.

IME its just confusing and does not add any value, you either broke the contract or you didn't. I find that developers use minor and patch interchangeably on a whim and as a result they might as well be one number.

Well and I don't use 0 because 0 means nothing and if there is nothing why deploy.

Anyways can we just parse the version match which works fine but use the original value not the semverfied one? Without starting a religious war maybe?

Helm support

Nice stuff.
I would love to see the Helm support e.g.:
helm upgrade release_name helm_chart --reuse-values --set imageTag="0.1.11"

Add notification support for Hipchat

Hi,

in my company we just started to use keel (it's a super tool!) and we need support for hipchat. I will implement notifications and later approvals for hipchat. I will provide a PR for this, maybe it can be helpful for someone else.

Native Webhook updates all helm releases even if keel isn't enabled

I currently have 3 namespaces for the an app - dev, qa, prod

I have enabled keel on the dev namespace via helm. When I went to POST a native webhook payload to keel it found that the qa and prod helm-releases matched

time="2017-10-12T07:40:09Z" level=info msg="provider.kubernetes: processing event" registry= repository=docker.artifactory.XXXXXX.com/user/test-proj tag=1.0-SNAPSHOT-develop
time="2017-10-12T07:40:09Z" level=info msg="provider.helm: processing event" registry= repository=docker.artifactory.XXXXXX.com/user/test-proj tag=1.0-SNAPSHOT-develop
time="2017-10-12T07:40:09Z" level=info msg="provider.kubernetes: no plans for deployment updates found for this event" image=docker.artifactory.XXXXXX.com/user/test-proj tag=1.0-SNAPSHOT-develop
time="2017-10-12T07:40:10Z" level=info msg="provider.helm: impacted release container found" parsed_image="docker.artifactory.XXXXXX.com/user/test-proj:1.0-SNAPSHOT-develop" policy=force target_image=docker.artifactory.XXXXXX.com/user/test-proj target_image_tag=1.0-SNAPSHOT-develop
time="2017-10-12T07:40:10Z" level=info msg="provider.helm: impacted release container found" parsed_image="docker.artifactory.XXXXXX.com/user/test-proj:1.0-RC2-release-1.0" policy=force target_image=docker.artifactory.XXXXXX.com/user/test-proj target_image_tag=1.0-SNAPSHOT-develop
time="2017-10-12T07:40:10Z" level=info msg="provider.helm: impacted release container found" parsed_image="docker.artifactory.XXXXXX.com/user/test-proj:1.0-RC2-release-1.0" policy=force target_image=docker.artifactory.XXXXXX.com/user/test-proj target_image_tag=1.0-SNAPSHOT-develop
time="2017-10-12T07:40:10Z" level=info msg="provider.helm: images do not match, ignoring" parsed_image_name="index.docker.io/karolisr/keel:0.5.0-rc.1" target_image_name=docker.artifactory.XXXXXX.com/user/test-proj
time="2017-10-12T07:40:13Z" level=info msg="provider.helm: release updated" release=project-dev version=8
time="2017-10-12T07:40:16Z" level=info msg="provider.helm: release updated" release=project-qa version=3
time="2017-10-12T07:40:19Z" level=info msg="provider.helm: release updated" release=project-prod version=3

Record the version change when updating deployment

When we create a deployment with
kubectl create -f <file.yaml> --record
it records the deployment history, which shows up when we do a
kubectl rollout history deployments/my-deployment

Right now, with Keel, it seems to update the same command everytime the revision gets bumped. for e.g. in my case, I started with image revision 1.0.0 , updated to 1.0.1, which updated my deployment but on displaying the rollout history, it shows like

emerald-2:k8s raja$ kubectl rollout history deployment/apache-deployment
deployments "apache-deployment"
REVISION	CHANGE-CAUSE
1		kubectl create --filename=httpd/nginx-deployment.yaml --record=true
2		kubectl create --filename=httpd/nginx-deployment.yaml --record=true

It looks like my original command has been repeated for the second revision. But if you look at https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#checking-rollout-history-of-a-deployment, it looks like when you can set the image using the "set image" command (Not sure what keel uses), it records it accordingly. This helps when doing rollbacks so we know the history when rolling back.

Would it be possible to do this so that the history becomes more clearer when using "rollout history"

cleanup logging

Currently Keel is logging way too much. It helps to understand how deployments/releases are being updated and hunt for bugs, but I want this to change before marking version 1.0.0

0.0.0 instead of latest

Hi!

I run private registry and don't use semver. I want to deploy images with tag latest with polling (no webhooks required).

CI also tags every image with build number for example, ${BUILD_ID}=75)

Deployment:

metadata:
  name: myapp
  labels:
    keel.sh/policy: force
    keel.sh/trigger: poll
  annotations:
    keel.sh/pollSchedule: "@every 1m"
spec:
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - image: myregistry.domain/myapp:latest
          imagePullPolicy: Always
          name: myapp
...

When new image is pushed to registry there's an entry in the log:

time="2017-09-05T12:54:26Z" level=info msg="provider.kubernetes: reseting image for force pull..." deployment=myapp image="myregistry.domain/myapp:0.0.0" namespace=default

As a result the image cannot be pulled with ErrImagePull error.

Keel image version is 0.4.7

Could you kindly help me with that?

How often to scan for docker images changes

an option to set how often to scan docker repository for image changes

Repo URL gets cut off

I have my keel: settings in values.yaml.:

keel:
  policy: all
  trigger: poll
  pollSchedule: "@every 5m"
  images:
    - repository: "quay.io/repo/app"
      tag: image.dockerTag

In the keel pod logs I see:

kube-system keel-keel-3958146990-5zn02 keel 2017/09/25 18:47:06 warning: destination for settings is a table. Ignoring non-table value <nil>
kube-system keel-keel-3958146990-5zn02 keel time="2017-09-25T18:47:06Z" level=error msg="provider.helm: failed to parse image" error="io/repo/app is not a value" repository_name=quay.io/repo/app repository_tag=image.dockerTag

String encoding issue

As mentioned in Slack, there appears to be a string encoding issue preventing pubsub subscriptions working:


2017-07-02T01:59:45.013498814Z time="2017-07-02T01:59:45Z" level=info msg="Keel starting.." arch=amd64 build_date=2017-07-01T135453Z go_version=go1.8.3 os=linux revision=2bb80bc version=2bb80bc 
2017-07-02T01:59:45.014058176Z time="2017-07-02T01:59:45Z" level=info msg="provider.kubernetes: using in-cluster configuration" 
2017-07-02T01:59:45.019152969Z time="2017-07-02T01:59:45Z" level=info msg="webhook trigger server starting..." port=9300 
2017-07-02T01:59:45.376872748Z time="2017-07-02T01:59:45Z" level=info msg="trigger.pubsub: subscribing for events..." subscription="keel-stemnapp-gcr.io%2Fstemnapp" topic="gcr.io%2Fstemnapp" 
2017-07-02T02:27:43.950750674Z time="2017-07-02T02:27:43Z" level=error msg="trigger.pubsub: got error while subscribing" error="rpc error: code = NotFound desc = Resource not found (resource=keel-stemnapp-gcr.io/stemnapp)." 
2017-07-02T02:27:43.950797682Z time="2017-07-02T02:27:43Z" level=error msg="trigger.pubsub.manager: failed to create subscription" error="rpc error: code = NotFound desc = Resource not found (resource=keel-stemnapp-gcr.io/stemnapp)." gcr_uri="gcr.io%2Fstemnapp" subscription_name="keel-stemnapp-gcr.io%2Fstemnapp" 
2017-07-02T02:27:47.012360626Z time="2017-07-02T02:27:47Z" level=error msg="trigger.pubsub.manager: failed to create subscription" error="failed to create subscription keel-stemnapp-gcr.io%2Fstemnapp, error: %!s(<nil>)" gcr_uri="gcr.io%2Fstemnapp" subscription_name="keel-stemnapp-gcr.io%2Fstemnapp" 
2017-07-02T02:28:45.422172104Z time="2017-07-02T02:28:45Z" level=info msg="trigger.pubsub: subscribing for events..." subscription="keel-stemnapp-gcr.io%2Fstemnapp" topic="gcr.io%2Fstemnapp" ```

Invalid Semantic Version

This seems like one of those cases when the latest tag should be handled gracefully, not an error thrown:

2017-07-11T01:48:40.268550511Z time="2017-07-11T01:48:40Z" level=warning msg="trigger.pubsub: failed to get name and version from image" action=INSERT error="Invalid Semantic Version" tag="gcr.io/stemnapp/alpine-website:latest" 
2017-07-11T01:48:41.821102919Z time="2017-07-11T01:48:41Z" level=info msg="provider.kubernetes: processing event" registry= repository="gcr.io/stemnapp/alpine-website" tag=v0.0.4

fix bug with 2 or more images to update in a single deployment

Drain feature

Hi,

I know this tool is meant to be kept simple, but a 'task drain' feature seems like a basic necessity of any deployment tool. When rolling out a new container, the existing one might be mid processing and not ready to end. This feature would gracefully request the container stop processing, then wait on it to complete.

To solve this issue I propose two apis, a deregister api and an isDrained api. The deregister api requests that the container gracefully shutdown, and the isDrained api returns whether or not is is finished processing. There would be a drainTimeout which would allow the task to fail or proceed forcefully after a certain amount of time.

Does this sound like something you'd benefit from? I know I (and likely others) would.

Index out of range

Logs from an unsuccessful run:


2017-07-02T02:39:46.948903643Z time="2017-07-02T02:39:46Z" level=info msg="Keel starting.." arch=amd64 build_date=2017-07-01T135453Z go_version=go1.8.3 os=linux revision=2bb80bc version=2bb80bc 
2017-07-02T02:39:46.94955744Z time="2017-07-02T02:39:46Z" level=info msg="provider.kubernetes: using in-cluster configuration" 
2017-07-02T02:39:46.954138278Z time="2017-07-02T02:39:46Z" level=info msg="webhook trigger server starting..." port=9300 
2017-07-02T02:39:47.246839657Z time="2017-07-02T02:39:47Z" level=info msg="trigger.pubsub: subscribing for events..." subscription="keel-stemnapp-gcr.io%2Fstemnapp" topic="gcr.io%2Fstemnapp" 
2017-07-02T02:39:48.032452458Z time="2017-07-02T02:39:48Z" level=info msg="provider.kubernetes: processing event" repository="gcr.io/stemnapp/alpine-api" 
2017-07-02T02:39:48.032490719Z time="2017-07-02T02:39:48Z" level=warning msg="trigger.pubsub: failed to get name and version from image" action=INSERT error="Invalid Semantic Version" tag="gcr.io/stemnapp/alpine-api:latest" 
2017-07-02T02:39:48.045175015Z time="2017-07-02T02:39:48Z" level=info msg="provider.kubernetes: keel policy found, checking deployment..." labels=map[keel.observer/policy:all role:api] name=api namespace=default policy=all 
2017-07-02T02:39:48.045344518Z time="2017-07-02T02:39:48Z" level=info msg="provider.kubernetes: checking image" image="gcr.io/stemnapp/alpine-api" name=api namespace=default parsed_image_name="gcr.io/stemnapp/alpine-api" policy=all target_image_name="gcr.io/stemnapp/alpine-api" target_tag=v0.0.825 
2017-07-02T02:39:48.047617496Z panic: runtime error: index out of range
2017-07-02T02:39:48.04766674Z 
2017-07-02T02:39:48.047718926Z goroutine 10 [running]:
2017-07-02T02:39:48.047885113Z github.com/rusenask/keel/util/version.GetVersionFromImageName(0xc4206e2140, 0x1a, 0xc4201530e0, 0x1a, 0xc4203dd001)
2017-07-02T02:39:48.047974208Z 	/go/src/github.com/rusenask/keel/util/version/version.go:42 +0xa6
2017-07-02T02:39:48.04818757Z github.com/rusenask/keel/provider/kubernetes.(*Provider).impactedDeployments(0xc42032df80, 0xc4203d7920, 0xc420153230, 0x25, 0x25, 0xc420153230, 0xc4205a15c0)
2017-07-02T02:39:48.048282215Z 	/go/src/github.com/rusenask/keel/provider/kubernetes/kubernetes.go:178 +0xe43
2017-07-02T02:39:48.048464159Z github.com/rusenask/keel/provider/kubernetes.(*Provider).processEvent(0xc42032df80, 0xc4203d7920, 0x1, 0x1, 0xc4201531d0, 0xc420020f38, 0xc420020e98)
2017-07-02T02:39:48.048561704Z 	/go/src/github.com/rusenask/keel/provider/kubernetes/kubernetes.go:82 +0x4f
2017-07-02T02:39:48.048717174Z github.com/rusenask/keel/provider/kubernetes.(*Provider).startInternal(0xc42032df80, 0x7c9b4b, 0xc42004b980)
2017-07-02T02:39:48.048724565Z 	/go/src/github.com/rusenask/keel/provider/kubernetes/kubernetes.go:66 +0x27c
2017-07-02T02:39:48.048728564Z github.com/rusenask/keel/provider/kubernetes.(*Provider).Start(0xc42032df80, 0x20459e0, 0x0)
2017-07-02T02:39:48.048731942Z 	/go/src/github.com/rusenask/keel/provider/kubernetes/kubernetes.go:51 +0x2b
2017-07-02T02:39:48.048735349Z created by main.setupProviders
2017-07-02T02:39:48.048738529Z 	/go/src/github.com/rusenask/keel/main.go:111 +0xf9```

Feature Request: Optionally always pull image on webhook

I often work with small projects that don't use versioning to keep track of docker containers, and it would be nice if I could get keel to force-pull the image instead of checking if there's a new version. Perhaps adding a parameter to keel.observer/policy, like always, would just pull the image without checks upon receiving a webhook.

A more advanced form of this would be determining whether or not to update the image based on SHA256 hash, but I would be happy with the former if it could be implemented faster.

EDIT: I just noticed this in util/version/version.go:

func ShouldUpdate(current *types.Version, new *types.Version, policy types.PolicyType) (bool, error) {
		if policy == types.PolicyTypeForce {
			return true, nil
…

I believe this might have been what i was asking about. ~~I'm assuming this is in the 0.2.4 image, so i'll update to that, adjust my labels and try again.~~ Will build image manually and test again.

Helm chart upgrades

I see keel currently supports updating the values in a helm chart to bump the version of the docker container. It would be useful if keel could upgrade the actual chart itself as upgrading the docker container works in some cases, but you lose other options like running init jobs from helm chart upgrades.

Issue: using build number instead of semver

We're using CI build numbers as version numbers for our internal applications, and tag our images accordingly (e.g. application:42 for build number 42). However, keel will try to parse this raw version number as a semver number, i.e. 42.0.0.

time="2017-07-30T15:32:37Z" level=info msg="provider.kubernetes: checking image" image="application:42.0.0" name=password namespace=default parsed_image_name="index.docker.io/application:42.0.0" policy=force target_image_name=application target_tag=42.0.0

Would it be possible to have proper support for build numbers as a version tag?

Images with similar names (starts_with) do not get updated?

We have two repos, shifts and shifts_api. shifts_api is updated fine, but shifts is not. I believe this is due to shifts receiving the webhook fine but resolving to shifts_api which then refuses to update since the images don't match. Logs included below.

time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes: processing event" registry= repository=ustwo/shifts tag=latest
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes.checkVersionedDeployment: keel policy found, checking deployment..." labels="map[app:shifts keel.sh/policy:force]" name=shifts namespace=default policy=force
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes: checking image" image="ustwo/shifts:latest" name=shifts namespace=default parsed_image_name="index.docker.io/ustwo/shifts:latest" policy=force target_image_name=ustwo/shifts target_tag=latest
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes: impacted deployment container found" parsed_image="index.docker.io/ustwo/shifts:latest" policy=force raw_image_name="ustwo/shifts:latest" target_image=ustwo/shifts target_image_tag=latest
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes.checkVersionedDeployment: keel policy found, checking deployment..." labels="map[app:shifts-api keel.sh/policy:force]" name=shifts-api namespace=default policy=force
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes: checking image" image="ustwo/shifts_api:latest" name=shifts-api namespace=default parsed_image_name="index.docker.io/ustwo/shifts_api:latest" policy=force target_image_name=ustwo/shifts target_tag=latest
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes: images do not match, ignoring" parsed_image_name="index.docker.io/ustwo/shifts_api:latest" target_image_name=ustwo/shifts
time="2017-08-09T14:18:24Z" level=info msg="provider.kubernetes: reseting image for force pull..." deployment=shifts image="ustwo/shifts:0.0.0" namespace=default
time="2017-08-09T14:18:26Z" level=info msg="provider.kubernetes: delta changed applied: ustwo/shifts:latest"

GCR PubSub Subscriber only sees one tag when multiple are pushed

Hey there,

We're currently using keel with Google Container Registry (GCR) + PubSub subscription and the helm provider for deployments. We're using the Google Cloud Container Builder (GCCB) to build our images and push them to GCR. In our builds, we create an image with the SemVer tag and one with the name of the git branch the image was built from. We sometimes cut pre-release off branches to facilitate testing of more experimental changes, which is why we've added support for tagging with either.

When these images are being pushed to GCR, we were noticing that keel seemed to only receive a message for one of the images being pushed (the one with the git branch and not the SemVer tag). This was causing keel to, rightfully, refuse to deploy the image:

time="2017-07-25T23:15:36Z" level=debug msg="trigger.pubsub: got message" action=INSERT image_name="dd-engineering/identity:master" tag=master 
time="2017-07-25T23:15:36Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/dd-engineering/identity tag=master 
time="2017-07-25T23:15:36Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/dd-engineering/identity tag=master 
time="2017-07-25T23:15:36Z" level=info msg="provider.kubernetes: no impacted deployments found for this event" image=gcr.io/dd-engineering/identity tag=master 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
2017/07/25 23:15:36 warning: destination for Annotations is a table. Ignoring non-table value <nil>
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" 
time="2017-07-25T23:15:36Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version"

That was when the GCCB configuration for which images to push looked like this:

images:
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_TAG}'
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_BRANCH}'

Switching it to this changed the message delivery so that keel could see the proper tag:

images:
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_BRANCH}'
  - 'gcr.io/$PROJECT_ID/identity:${_GIT_TAG}'

I've not done a deep-enough dive to know whether this is a parsing issue in keel or whether the GCR PubSub integration doesn't contain both tags.

optional configuration of "change windows"

It would be nice to have a way to tell Keel to only update deployments during certain time of the day. Example: webhooks/pubsub message hits keel on 9:00pm but it waits till 03:00am and only then performs the update

Rollback example?

I was wondering if there were examples of common workflows, especially with regards to rollbacks. I was contemplating doing this given keel with a kubernetes provider, a policy of force (one policy on docker tag staging and one on tag prod), and polling a private docker repository:

docker push myregistry.com/myimage:v1.2.3
docker tag myregistry.com/myimage:v1.2.3 myregistry.com/myimage:staging
docker push myregistry.com/myimage:staging

Keel then deploys the app to my kubernetes staging cluster, and then keel will send a notification via webhook to my staging jenkins to run integration tests. If these, succeed, then I do this:

docker tag myregistry.com/myimage:v1.2.3 myregistry.com/myimage:prod
docker push myregistry.com/myimage:prod

Keel then deploys the app to my kubernetes prod cluster.

If we notice problems in production (increased error rates, etc.) with 1.2.3 and want to roll back to 1.2.2, I think I then can manually do this:

docker tag myregistry.com/myimage:v1.2.2 myregistry.com/myimage:prod
docker push myregistry.com/myimage:prod

Keel would notice the tag changes, and deploy things back to how they had been before. Is that an accurate understanding? Are there better/recommended ways to do this?

Add RBAC to Helm chart

secrets are not found

Hi together,
I'm currently facing some problem when trying to use keel.
I have the following setup.

private repository in quay.io
robot account set up in quay.io
secret.yaml file provided by quay.io deployed to kubernetes cluster

Then I installed an own package via helm and set the imagePullSecret in the template file of the deployment.
When I now push a new image to the quay.io repo keel should pull automatically the new version but currently it fails to authenticate against quay.io with the error message

level=info msg="secrets.defaultGetter.lookupSecrets: no secrets for image found" image=quay.io/group/myImage namespace=default pod_selector="app=myImageApp,release=dangling-aardvark" pods_checked=0 provider=helm registry=quay.io

Is there another place where I should reference the secret or am I using it completely wrong?

Thank you in advance.

Private docker registries support for polling

400 Errors talking to gcr.io

Im a newbie trying out Keel, so it could be something that Ive done, but havent been able to find what it is after looking at all the documentation available.

I have a GKE cluste running 1.7.8 and Ive setup Keel as mentioned in the install guide. On one of my deployments which has the image hosted on gcr.io on the same project, I set up

  keel.sh/policy: major
  keel.sh/trigger: poll

to see if Polling would work. Once the setup was run and doing a kubectl log -f -n kube-system, I get the following statements repeated.

time="2017-12-02T08:42:29Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Head https://gcr.io/v2/upheld-flow-183107/php/manifests/5.0.0: http: non-successful response (status=400 body=\"\")" image="gcr.io/upheld-flow-183107/php:5.0.0"
time="2017-12-02T08:42:29Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Head https://gcr.io/v2/upheld-flow-183107/php/manifests/5.0.0: http: non-successful response (status=400 body=\"\")" image_name="gcr.io/upheld-flow-183107/php:5.0.0" registry_username=
time="2017-12-02T08:42:29Z" level=error msg="trigger.poll.manager: failed to start watching repository" error="Head https://gcr.io/v2/upheld-flow-183107/php/manifests/5.0.0: http: non-successful response (status=400 body=\"\")" image="gcr.io/upheld-flow-183107/php:5.0.0" schedule="@every 1m"

It looks like its trying to authenticate but its failing but Im not able to see how to setup access for it (if its required).

My original deployment is running well, but I was expecting a postive log about being able to inform the deployment to slack notifier, but I have the above repeating every minute (default polling interval)

Can someone point as to what is wrong in this setup?

Environment:
Keel : 0.4.7
GKE running 1.7.8-gke.0

add dockerhub integration

add dockerhub integration
test it

Using a gcp service account credentials

Good morning,

I run keel with this configuration:

DEBUG=1
GOOGLE_APPLICATION_CREDENTIALS=points to a service account credentials file
POLL=1
PUBSUB=1
PROJECT_ID=my registry project id on gcp
SLACK_TOKEN=a bot token from slack
SLACK_CHANNELS=general
SLACK_APPROVALS_CHANNEL=general

On startup keel logs the following:

time="2018-01-06T12:46:23Z" level=info msg="extension.notification: sender registered" name=hipchat
time="2018-01-06T12:46:23Z" level=info msg="extension.notification: sender registered" name=slack
time="2018-01-06T12:46:23Z" level=info msg="extension.notification: sender registered" name=webhook
time="2018-01-06T12:46:23Z" level=info msg="bot: registered" name=slack
time="2018-01-06T12:46:23Z" level=info msg="keel starting..." arch=amd64 build_date=2018-01-04T104054Z go_version=go1.9.2 os=linux revision=1ef066f version=0.5.2
time="2018-01-06T12:46:23Z" level=info msg="extension.notification.slack: sender configured" channels="[general]" name=slack
time="2018-01-06T12:46:23Z" level=info msg="notificationSender: sender configured" sender name=slack
time="2018-01-06T12:46:23Z" level=info msg="provider.kubernetes: using in-cluster configuration"
time="2018-01-06T12:46:23Z" level=info msg="provider.defaultProviders: provider 'kubernetes' registered"
time="2018-01-06T12:46:23Z" level=info msg="webhook trigger server starting..." port=9300
time="2018-01-06T12:46:24Z" level=info msg="trigger.poll.manager: polling trigger configured"
time="2018-01-06T12:47:19Z" level=debug msg="performing scan"
time="2018-01-06T12:47:24Z" level=debug msg="performing scan"

It doesn't look like anything gets reported by trigger.pubsub so I am wondering if it correctly hooked to it and how I could confirm that. The service account tied to the GOOGLE_APPLICATION_CREDENTIALS was granted roles/pubsub.editor and roles/storage.objectViewer (required for read on gcr).

I see no subscription on GCP's pubsub side.

Can someone help me troubleshoot this?

Failing to auth against Jfrog Artifactory for private repositories

I'm trying to do use Artifactory as docker store with polling using the Kubernetes provider and it seems the user is not coming through.

Its picking up the secret OK and shows the correct username in the log, but somehow does not seem to be sending it on correctly. I've tried using the api key and user/pass that are validated on the Artifactory side.

Using 90515df8f4b5bc95042f1bbeee05221ae6f27b8f2ded55286063d2a8e4b901c1
Kube 1.5.7

Created the secrets using

kubectl create secret docker-registry myregistrykey --docker-server=https://index.docker.io/v1/ --docker-username=<user> --docker-password=<pass> --docker-email=<email>

But I'm sure that is working fine

time="2017-08-02T02:23:47Z" level=debug msg="performing scan" 
time="2017-08-02T02:23:47Z" level=info msg="secrets.defaultGetter: secret looked up successfully" image=company-name-docker.jfrog.io/app-report-services namespace=delivery provider=kubernetes registry=company-name-docker.jfrog.io 
time="2017-08-02T02:23:47Z" level=info msg="registry client: getting digest" registry="https://company-name-docker.jfrog.io" repository=app-report-services tag=10.29 
time="2017-08-02T02:23:47Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://company-name-docker.jfrog.io/v2/: http: non-successful response (status=403 body=\"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 403,\\n    \\\"message\\\" : \\\"Forbidden\\\"\\n  } ]\\n}\")" image="company-name-docker.jfrog.io/app-report-services:10.29" 
time="2017-08-02T02:23:47Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://company-name-docker.jfrog.io/v2/: http: non-successful response (status=403 body=\"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 403,\\n    \\\"message\\\" : \\\"Forbidden\\\"\\n  } ]\\n}\")" image_name="company-name-docker.jfrog.io/app-report-services:10.29" registry_username=namebuild 
time="2017-08-02T02:23:47Z" level=error msg="trigger.poll.manager: failed to start watching repository" error="Get https://company-name-docker.jfrog.io/v2/: http: non-successful response (status=403 body=\"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 403,\\n    \\\"message\\\" : \\\"Forbidden\\\"\\n  } ]\\n}\")" image="company-name-docker.jfrog.io/app-report-services:10.29" schedule="@every 1m"

util/version: pre-release tags are not taken in to consideration in ShouldUpgrade()

In my organization we're working on using SemVer for all of our releases, with the use of pre-release tags to test experimental builds. For pre-release builds, we increment the patch and then add the pre-release information at the end. So if v1.2.2 is the last tag, the pre-release would be v1.2.3-preNNNNNN.

The problem is that keel does not currently inspect the information about the PreRelease to understand whether it's a "proper release"[1]. This means that if you have your policy set to patch, a pre-release of the next patch version will be deployed. To give a quick overview with each policy type there is a different combination of pre-release tags which someone may want to deploy:

v1.2.2 => v1.2.3-pre
v1.2.3 => v1.3.0-pre
v1.2.3 => v2.0.0-pre
v1.2.3-pre => v1.3.0-pre
v1.2.3-pre => v1.3.0

Attached to the bottom of this issue is a patch with a unit test showing the failure mode that made me realize this was a potential issue. [2]

To solving the problem, it seems there are two ways to solve this problem within keel:

Add more policies (e.g., PolicyTypePatchPreRelease, PolicyTypeMinorPreRelease, etc.) and do the additional checks
Introduce a prelreasePolicy value that can contain the same values as policy does now (and maybe rename policy to releasePolicy). We could then have a matrix of behaviors based on the releasePolicy and the prereleasePolicy.

The second approach feels like the most powerful, and could allow you to do something like have a releasePolicy: minor so you get all of the minor + patch updates, but a prereleasePolicy: patch so that you only run experimental patch builds. This might be useful for something like a QA environment.

I think one of the easier ways to tackle this, since it's already a numerical value, to use it as bitwise flags. Policies could then be created by doing something like types.PolicyTypePreReleasePatch & types.PolicyTypeMinor. I'd be happy to take on implementing it once a good design is figured out.

[1] I quote proper release, because this could be up to who you ask.
[2]

diff --git a/util/version/version_test.go b/util/version/version_test.go
index f3c7d68..d172f8f 100644
--- a/util/version/version_test.go
+++ b/util/version/version_test.go
@@ -173,6 +173,16 @@ func TestShouldUpdate(t *testing.T) {
 			wantErr: false,
 		},
 		{
+			name: "patch pre-release, policy patch",
+			args: args{
+				current: &types.Version{Major: 1, Minor: 4, Patch: 5},
+				new:     &types.Version{Major: 1, Minor: 4, Patch: 6, PreRelease: "pre-20170728121212"},
+				policy:  types.PolicyTypePatch,
+			},
+			want:    false,
+			wantErr: false,
+		},
+		{
 			name: "patch increase, policy ptach",
 			args: args{
 				current: &types.Version{Major: 1, Minor: 4, Patch: 5},

Helm Provider: rpc error (timeout) during update of release

I got a timeout during the update of a helm release:

time="2017-07-25T23:39:04Z" level=error msg="provider.helm: failed to apply plan" error="rpc error: code = Unknown desc = timed out waiting for the condition" name=is-app namespace=default

time="2017-07-25T23:38:33Z" level=debug msg="trigger.pubsub: got message" action=INSERT image_name="dd-engineering/identity:1.8.1" tag=1.8.1 
time="2017-07-25T23:38:33Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/dd-engineering/identity tag=1.8.1 
time="2017-07-25T23:38:33Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/dd-engineering/identity tag=1.8.1 
time="2017-07-25T23:38:33Z" level=info msg="provider.kubernetes: no impacted deployments found for this event" image=gcr.io/dd-engineering/identity tag=1.8.1 
image.tag
gcr.io/dd-engineering/identity
time="2017-07-25T23:38:33Z" level=info msg="provider.helm: current image version" container_image=gcr.io/dd-engineering/identity current_version=1.6.1 name=is-app namespace=default policy=minor 
time="2017-07-25T23:38:33Z" level=info msg="provider.helm: checked version, deciding whether to update" container_image=gcr.io/dd-engineering/identity current_version=1.6.1 name=is-app namespace=default new_version=1.8.1 policy=minor should_update=true 
time="2017-07-25T23:38:33Z" level=info msg="provider.helm: impacted release tags found" container_image=gcr.io/dd-engineering/identity policy=minor target_image=gcr.io/dd-engineering/identity target_image_tag=1.8.1

It seems like the code around here https://github.com/rusenask/keel/blob/28553b3e9db8076de8c5b0d50eb664c4b6f7e92f/provider/helm/helm.go#L269-L285 (haven't dug much deeper than that I'm afraid)

Make Slack channels configurable

Hi!
It may be useful to configure Slack channels (for approvals and notifications) for each deployment.

in pubsub.manager.checkDeployment refactor subscribtion

Problem inside that function is when dealing with deployments that have sidecar containers. It results in double subscription if both containers use images from the same gcr

Watch myimage:latest without any hooks

Hi,

Does Keel supports polling the registry let's say every minute to check if the sha256 of an image has changed?

Many devs wants that feature, I still have a hard time to understand if that's supported or not.

Best,

if two Keel instances subscribe to one topic - one will not receive messages

Basically subscription types needs to be changed.

Edit: after some investigation it appears that we need to:

update pubsub library
set unique subscription names per Keel installation

Quay image pull - invalid bearer token

2017/10/02 13:24:17 registry failed ping request, error: Get https://quay.io/v2/: http: non-successful response (status=401 body="{\"error\": \"Invalid bearer token format\"}")

Secret was created using kubectl create secret docker-registry and k8s has no problem pulling the image manually using that secret.

Exclusive version updates

Is it possible to have a policy to update on:

only patch changes:

  from 1.0.1 => 1.0.2 = update
  from 1.0.1 => 1.2.0 = do not update
  from 1.2.0 => 1.2.1 = update

only minor changes:

  form 1.0.1 => 1.0.2 = do not update
  form 1.0.1 => 1.2.0 = update
  form 1.2.0 => 1.2.1 = do not update
  form 1.2.1 => 2.0.0 = do no update

only mayor changes:

  from 1.0.0 => 1.0.1 = do not update
  from 1.0.0 => 1.1.0 = do not update
  from 1.1.0 => 2.0.0 = update
  from 2.0.0 => 2.0.1 = do not update

But why? I have 3 environments, development > staging > production, and I'm thinking I can increment the patch version for development releases, then the minor version for staging releases, and the mayor for production.

But I need, for example, production exclusively only watch mayor changes, staging only watch minor changes, and development for patch updates.

You you think I can set this up? good or bad idea? can somebody point me into the right direction to implement this?

Thanks in advance.

Docker credentials were not found among secrets

Hello, I can pull images from my privates registry, but keel (v0.47) can't connect on it. The error raised is time="2017-11-28T17:35:44Z" level=warning msg="secrets.defaultGetter.lookupSecrets: docker credentials were not found among secrets" image=private.registry.com/myproject/example namespace=dev provider=kubernetes registry=private.registry.com secrets="[private-registry]" My secrets .dockercfg looks like

{
    "gcr.myproject.com": {
        "auth": "base64sddfsdf"
    },
    "ucr.myproject.io": {
        "auth": "base64sfdsdf"
    }
}

The release 0.43 explain we should use secrets created from kubectl create secret docker-registry myregistrykey --docker-server=https://index.docker.io/v1/ --docker-username=<user> --docker-password=<pass> --docker-email=<email> so it should looks like

{"https://index.docker.io/v1/":{"username":"user","password":"pass","email":"email","auth":"dXNlcjpwYXNz"}}

And in the code https://github.com/keel-hq/keel/blob/master/secrets/secrets.go#L213-L226 we can see it's possible to extract credentials from auth. So, is it mandatory to have username and password in secret dockercfg to allow keel to contact the registry for a poll or is it my dockercfg format which is the problem ?

Here the answer on slack:

karolis [12:28 PM]
Hello, there are several (I think 3) ways credentials can be stored for registries
currently it can either work with username/password or base 64 encoded auth
if you could create an issue with an example command on how you are creating secret I could then try to reproduce it
your dockercfg should work 😐

You can found everything to reproduce the case on this gist.
https://gist.github.com/vonglasow/efcf09821b6d86b1c0b85bbbbc695476

Unsupported semver format

Logs after processing a container tagged with v0.0.824.

2017-07-01T12:18:47.845612596Z time="2017-07-01T12:18:47Z" level=info msg="Keel starting.." arch=amd64 build_date=2017-06-27T144643Z go_version=go1.8.3 os=linux revision=d1a9752 version=d1a9752 
2017-07-01T12:18:47.845948506Z time="2017-07-01T12:18:47Z" level=info msg="provider.kubernetes: using in-cluster configuration" 
2017-07-01T12:18:47.850662597Z time="2017-07-01T12:18:47Z" level=info msg="webhook trigger server starting..." port=9300 
2017-07-01T12:18:48.13538062Z time="2017-07-01T12:18:48Z" level=info msg="trigger.pubsub: subscribing for events..." subscription="keel-stemnapp-gcr.io%2Fstemnapp" topic="gcr.io%2Fstemnapp" 
2017-07-01T12:21:47.530981708Z time="2017-07-01T12:21:47Z" level=warning msg="trigger.pubsub: failed to get name and version from image" action=INSERT error="latest is not in dotted-tri format" tag="gcr.io/stemnapp/alpine-api:latest" 
2017-07-01T12:21:49.697335787Z time="2017-07-01T12:21:49Z" level=warning msg="trigger.pubsub: failed to get name and version from image" action=INSERT error="latest is not in dotted-tri format" tag="gcr.io/stemnapp/alpine-worker:latest" 
2017-07-01T12:46:28.075342977Z time="2017-07-01T12:46:28Z" level=warning msg="trigger.pubsub: failed to get name and version from image" action=INSERT error="latest is not in dotted-tri format" tag="gcr.io/stemnapp/alpine-api:latest" 
2017-07-01T12:46:29.426435699Z time="2017-07-01T12:46:29Z" level=warning msg="trigger.pubsub: failed to get name and version from image" action=INSERT error="strconv.ParseInt: parsing "v0": invalid syntax" tag="gcr.io/stemnapp/alpine-api:v0.0.824"

time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes: processing event" registry= repository=some/repo tag=latest
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes.checkVersionedDeployment: keel policy found, checking deployment..." labels="map[app:shifts keel.sh/policy:force]" name=shifts namespace=default policy=force
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes: checking image" image="some/repo:latest" name=shifts namespace=default parsed_image_name="index.docker.io/some/repo:latest" policy=force target_image_name=some/repo target_tag=latest
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes: impacted deployment container found" parsed_image="index.docker.io/some/repo:latest" policy=force raw_image_name="some/repo:latest" target_image=some/repo target_image_tag=latest
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes.checkVersionedDeployment: keel policy found, checking deployment..." labels="map[keel.sh/policy:force app:shifts-api]" name=shifts-api namespace=default policy=force
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes: checking image" image="some/repo_api:latest" name=shifts-api namespace=default parsed_image_name="index.docker.io/some/repo_api:latest" policy=force target_image_name=some/repo target_tag=latest
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes: images do not match, ignoring" parsed_image_name="index.docker.io/some/repo_api:latest" target_image_name=some/repo
time="2017-08-09T13:34:37Z" level=info msg="provider.kubernetes: reseting image for force pull..." deployment=shifts image="some/repo:0.0.0" namespace=default
time="2017-08-09T13:34:39Z" level=info msg="provider.kubernetes: delta changed applied: some/repo:latest"