I'm currently testing lagoon-master with lagoon-kbd against a kubernetes cluster and found the following issue:
Lagoon-master creates a rabbitmq message like this:
{
"metadata": {
"name": "lagoon-build-qzzyhp",
"namespace": "lagoon"
},
"spec": {
"build": {
"type": "pullrequest",
"image": {},
"ci": "false"
},
"branch": {
"name": "pr-78"
},
"pullrequest": {
"headBranch": "pr-testing",
"headSha": "14627d2b6061ed1a1a1b8a692963145271a9a5d8",
"baseBranch": "master",
"baseSha": "229ec3e57e220114e1c6ab60dde07a8ea12b873b",
"title": "Update system.site.yml - update",
"number": "78"
},
"project": {
"name": "drupal-example-test1-controller",
"gitUrl": "[email protected]:amazeeio/drupal-example.git",
"uiLink": "https://ui-lagoon-master.ch.amazee.io/projects/drupal-example-test1-controller/drupal-example-test1-controller-pr-78/deployments/lagoon-build-qzzyhp",
"environment": "pr-78",
"environmentType": "development",
"productionEnvironment": "master2",
"standbyEnvironment": "master",
"subfolder": "",
"routerPattern": "pr-78.drupal-example-test1-controller.test1.amazee.io",
"deployTarget": "test1.amazee.io",
"projectSecret": "7d6c71a7f475e53b96142e1ce7b141e80f8de59365e17b964a6085fd4bd96f73",
"registry": "registry.lagoon.svc:5000",
"monitoring": {
"contact": "",
"statuspageID": "null"
},
"variables": {
"project": "W3sibmFtZSI6IklOVEVSTkFMX1JFR0lTVFJZX1VSTCIsInZhbHVlIjoiaHR0cHM6Ly9oYXJib3ItbmdpbngtbGFnb29uLW1hc3Rlci5jaC5hbWF6ZWUuaW8iLCJzY29wZSI6ImludGVybmFsX2NvbnRhaW5lcl9yZWdpc3RyeSJ9LHsibmFtZSI6IklOVEVSTkFMX1JFR0lTVFJZX1VTRVJOQU1FIiwidmFsdWUiOiJyb2JvdCRkcnVwYWwtZXhhbXBsZS10ZXN0MS1jb250cm9sbGVyLTE2MDc5MTEyNTMiLCJzY29wZSI6ImludGVybmFsX2NvbnRhaW5lcl9yZWdpc3RyeSJ9LHsibmFtZSI6IklOVEVSTkFMX1JFR0lTVFJZX1BBU1NXT1JEIiwidmFsdWUiOiJleUpoYkdjaU9pSlNVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKbGVIQWlPakUyTlRFeE1URXlOVE1zSW1saGRDSTZNVFl3TnpreE1USTFNeXdpYVhOeklqb2lhR0Z5WW05eUxYUnZhMlZ1TFdSbFptRjFiSFJKYzNOMVpYSWlMQ0pwWkNJNk16SXhOVGM1TENKd2FXUWlPakV4TkRNc0ltRmpZMlZ6Y3lJNlczc2lVbVZ6YjNWeVkyVWlPaUl2Y0hKdmFtVmpkQzh4TVRRekwzSmxjRzl6YVhSdmNua2lMQ0pCWTNScGIyNGlPaUp3ZFhOb0lpd2lSV1ptWldOMElqb2lJbjFkZlEucG43c3VDTlBUYTl3bmhuY1J1cVozZEFIQ09ka3RTWXpQdmo5TFlzbHhFWm1hd25uMWFwSU9VZGdGVHZ0YVk1cDRwbGZTQlM0WlZndm81U0tHWEJIOWprSVFpWjFZVkx5dnZPOU43YWROVi1sakNTaXNaSDNGaGlmZHRQSUkySkxtUTV6QldOOUhSUjAzcmZKMkdCeDVndkd0Uy1aUkVsSFlFRUpaeWdTVXRvMGk3MzFPWmhmSGVlRTE1UW83cEh2Q2hxWkhhTXJsOS0wamZGN01RTTltNGN3MGNpWllUa0NhMkxFQkt6aWVvT1hnN3IweVlSd2ZFeGlGTWJ2bjBJZTV3YXFmZEU5UEZTR2FYWTRmcFJIX0Q3dlZFLXdBN3FmMGYyWktIS2s0cGY5dlpQd1JXLWRQU0t1RGl0blhycC1qZ3VzR1R6UEhTczFEdDlLZDhxb1dZYTNZbHVVRnVCcndUeDJndFdVb2g1NnRibjlHMGloTG5ybUwyZUNUUzRBRzNWbW1kUWRhUGN6WTZFejhGdVh6SW9UVHhmSUxjTlNHWGljM00yN1hGX2RON2d0OWpQaTVfWEJMekNWYVRQNHZ6TDN4eVByVHV5dWNseWFVR0JkNl9GelRIeWRwNXlJczVRS2czREhCSm9ielB4ZzNCZEJqbTNXNmd5blh2bmNadWU5UmVqMmdHb3lNa0xPT21QS2VWcnhXNm1uWmptd2l4blR1cE5vVzA2Vktqek9ZRWRHenkwVmJKQnBjVk03Y2xCQ1BYb2x0TElGQ3hsRWVib3lzV1NYVGk4dk0zeXR4WEFfenlkeG5uNDVET3RVQkxBdENtUlFWSXJuYUpMNDFpSGtwZ0NtVU9yWW1zNFlkSWRhMVAzbXRhR3hSc0pEckI4RzVkZkFnelUiLCJzY29wZSI6ImludGVybmFsX2NvbnRhaW5lcl9yZWdpc3RyeSJ9XQ==",
"environment": "W10="
}
}
}
}
see the spec.pullrequest.number in there
but then the LagoonBuild object is created by lagoon-build-deploy and it disapears:
kind: LagoonBuild
apiVersion: lagoon.amazee.io/v1alpha1
metadata:
creationTimestamp: '2020-12-24T15:11:26Z'
finalizers:
- finalizer.lagoonbuild.lagoon.amazee.io/v1alpha1
generation: 2
labels:
lagoon.sh/buildStatus: Failed
lagoon.sh/controller: lagoon
managedFields:
- apiVersion: lagoon.amazee.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:finalizers':
.: {}
'v:"finalizer.lagoonbuild.lagoon.amazee.io/v1alpha1"': {}
'f:labels':
.: {}
'f:lagoon.sh/buildStatus': {}
'f:lagoon.sh/controller': {}
'f:spec':
.: {}
'f:branch':
.: {}
'f:name': {}
'f:build':
.: {}
'f:ci': {}
'f:type': {}
'f:gitReference': {}
'f:project':
.: {}
'f:deployTarget': {}
'f:environment': {}
'f:environmentType': {}
'f:gitUrl': {}
'f:key': {}
'f:monitoring':
.: {}
'f:statuspageID': {}
'f:name': {}
'f:productionEnvironment': {}
'f:projectSecret': {}
'f:registry': {}
'f:routerPattern': {}
'f:standbyEnvironment': {}
'f:uiLink': {}
'f:variables':
.: {}
'f:environment': {}
'f:project': {}
'f:promote': {}
'f:pullrequest':
.: {}
'f:baseBranch': {}
'f:baseSha': {}
'f:headBranch': {}
'f:headSha': {}
'f:title': {}
'f:status':
.: {}
'f:conditions': {}
manager: manager
operation: Update
time: '2020-12-24T15:11:44Z'
name: lagoon-build-qzzyhp
namespace: drupal-example-test1-controller-pr-78
resourceVersion: '132895455'
selfLink: >-
/apis/lagoon.amazee.io/v1alpha1/namespaces/drupal-example-test1-controller-pr-78/lagoonbuilds/lagoon-build-qzzyhp
uid: f969b554-bed7-4fa2-9c9f-70881db54e80
spec:
branch:
name: pr-78
build:
ci: 'false'
type: pullrequest
gitReference: ''
project:
deployTarget: test1.amazee.io
environment: pr-78
environmentType: development
gitUrl: '[email protected]:amazeeio/drupal-example.git'
monitoring:
statuspageID: 'null'
name: drupal-example-test1-controller
productionEnvironment: master2
projectSecret: 7d6c71a7f475e53b96142e1ce7b141e80f8de59365e17b964a6085fd4bd96f73
registry: 'registry.lagoon.svc:5000'
routerPattern: pr-78.drupal-example-test1-controller.test1.amazee.io
standbyEnvironment: master
uiLink: >-
https://ui-lagoon-master.ch.amazee.io/projects/drupal-example-test1-controller/drupal-example-test1-controller-pr-78/deployments/lagoon-build-qzzyhp
variables:
environment: W10=
project: >-
W3sibmFtZSI6IklOVEVSTkFMX1JFR0lTVFJZX1VSTCIsInZhbHVlIjoiaHR0cHM6Ly9oYXJib3ItbmdpbngtbGFnb29uLW1hc3Rlci5jaC5hbWF6ZWUuaW8iLCJzY29wZSI6ImludGVybmFsX2NvbnRhaW5lcl9yZWdpc3RyeSJ9LHsibmFtZSI6IklOVEVSTkFMX1JFR0lTVFJZX1VTRVJOQU1FIiwidmFsdWUiOiJyb2JvdCRkcnVwYWwtZXhhbXBsZS10ZXN0MS1jb250cm9sbGVyLTE2MDc5MTEyNTMiLCJzY29wZSI6ImludGVybmFsX2NvbnRhaW5lcl9yZWdpc3RyeSJ9LHsibmFtZSI6IklOVEVSTkFMX1JFR0lTVFJZX1BBU1NXT1JEIiwidmFsdWUiOiJleUpoYkdjaU9pSlNVekkxTmlJc0luUjVjQ0k2SWtwWFZDSjkuZXlKbGVIQWlPakUyTlRFeE1URXlOVE1zSW1saGRDSTZNVFl3TnpreE1USTFNeXdpYVhOeklqb2lhR0Z5WW05eUxYUnZhMlZ1TFdSbFptRjFiSFJKYzNOMVpYSWlMQ0pwWkNJNk16SXhOVGM1TENKd2FXUWlPakV4TkRNc0ltRmpZMlZ6Y3lJNlczc2lVbVZ6YjNWeVkyVWlPaUl2Y0hKdmFtVmpkQzh4TVRRekwzSmxjRzl6YVhSdmNua2lMQ0pCWTNScGIyNGlPaUp3ZFhOb0lpd2lSV1ptWldOMElqb2lJbjFkZlEucG43c3VDTlBUYTl3bmhuY1J1cVozZEFIQ09ka3RTWXpQdmo5TFlzbHhFWm1hd25uMWFwSU9VZGdGVHZ0YVk1cDRwbGZTQlM0WlZndm81U0tHWEJIOWprSVFpWjFZVkx5dnZPOU43YWROVi1sakNTaXNaSDNGaGlmZHRQSUkySkxtUTV6QldOOUhSUjAzcmZKMkdCeDVndkd0Uy1aUkVsSFlFRUpaeWdTVXRvMGk3MzFPWmhmSGVlRTE1UW83cEh2Q2hxWkhhTXJsOS0wamZGN01RTTltNGN3MGNpWllUa0NhMkxFQkt6aWVvT1hnN3IweVlSd2ZFeGlGTWJ2bjBJZTV3YXFmZEU5UEZTR2FYWTRmcFJIX0Q3dlZFLXdBN3FmMGYyWktIS2s0cGY5dlpQd1JXLWRQU0t1RGl0blhycC1qZ3VzR1R6UEhTczFEdDlLZDhxb1dZYTNZbHVVRnVCcndUeDJndFdVb2g1NnRibjlHMGloTG5ybUwyZUNUUzRBRzNWbW1kUWRhUGN6WTZFejhGdVh6SW9UVHhmSUxjTlNHWGljM00yN1hGX2RON2d0OWpQaTVfWEJMekNWYVRQNHZ6TDN4eVByVHV5dWNseWFVR0JkNl9GelRIeWRwNXlJczVRS2czREhCSm9ielB4ZzNCZEJqbTNXNmd5blh2bmNadWU5UmVqMmdHb3lNa0xPT21QS2VWcnhXNm1uWmptd2l4blR1cE5vVzA2Vktqek9ZRWRHenkwVmJKQnBjVk03Y2xCQ1BYb2x0TElGQ3hsRWVib3lzV1NYVGk4dk0zeXR4WEFfenlkeG5uNDVET3RVQkxBdENtUlFWSXJuYUpMNDFpSGtwZ0NtVU9yWW1zNFlkSWRhMVAzbXRhR3hSc0pEckI4RzVkZkFnelUiLCJzY29wZSI6ImludGVybmFsX2NvbnRhaW5lcl9yZWdpc3RyeSJ9XQ==
promote: {}
pullrequest:
baseBranch: master
baseSha: 229ec3e57e220114e1c6ab60dde07a8ea12b873b
headBranch: pr-testing
headSha: 14627d2b6061ed1a1a1b8a692963145271a9a5d8
title: Update system.site.yml - update
see that now spec.pullrequest.number is missing
the pod that is created to deploy PRs then looks like this:
kind: Pod
apiVersion: v1
metadata:
name: lagoon-build-qzzyhp
namespace: drupal-example-test1-controller-pr-78
selfLink: >-
/api/v1/namespaces/drupal-example-test1-controller-pr-78/pods/lagoon-build-qzzyhp
uid: 09b491ed-f686-4f37-b87a-52c9983527e6
resourceVersion: '132895453'
creationTimestamp: '2020-12-24T15:11:27Z'
labels:
lagoon.sh/buildName: lagoon-build-qzzyhp
lagoon.sh/controller: lagoon
lagoon.sh/jobType: build
annotations:
kubernetes.io/psp: eks.privileged
ownerReferences:
- apiVersion: lagoon.amazee.io/v1alpha1
kind: LagoonBuild
name: lagoon-build-qzzyhp
uid: f969b554-bed7-4fa2-9c9f-70881db54e80
spec:
volumes:
- name: lagoon-deployer-token-r5b7f
secret:
secretName: lagoon-deployer-token-r5b7f
defaultMode: 420
- name: lagoon-sshkey
secret:
secretName: lagoon-sshkey
defaultMode: 420
- name: default-token-mgnb7
secret:
secretName: default-token-mgnb7
defaultMode: 420
containers:
- name: lagoon-build
image: 'imagecache.amazeeio.cloud/amazeeio/kubectl-build-deploy-dind:v1.13.1'
env:
- name: SOURCE_REPOSITORY
value: '[email protected]:amazeeio/drupal-example.git'
- name: GIT_REF
- name: SUBFOLDER
- name: BRANCH
value: pr-78
- name: PROJECT
value: drupal-example-test1-controller
- name: ENVIRONMENT_TYPE
value: development
- name: ACTIVE_ENVIRONMENT
value: master2
- name: STANDBY_ENVIRONMENT
value: master
- name: PROJECT_SECRET
value: 7d6c71a7f475e53b96142e1ce7b141e80f8de59365e17b964a6085fd4bd96f73
- name: MONITORING_ALERTCONTACT
- name: BUILD_TYPE
value: pullrequest
- name: ENVIRONMENT
value: pr-78
- name: KUBERNETES
value: test1.amazee.io
- name: REGISTRY
value: 'registry.lagoon.svc:5000'
- name: ROUTER_URL
value: pr-78.drupal-example-test1-controller.test1.amazee.io
- name: CI
value: 'false'
- name: PR_HEAD_BRANCH
value: pr-testing
- name: PR_HEAD_SHA
value: 14627d2b6061ed1a1a1b8a692963145271a9a5d8
- name: PR_BASE_BRANCH
value: master
- name: PR_BASE_SHA
value: 229ec3e57e220114e1c6ab60dde07a8ea12b873b
- name: PR_TITLE
value: Update system.site.yml - update
- name: PR_NUMBER
value: "\0"
- name: LAGOON_PROJECT_VARIABLES
value: >-
[{"name":"INTERNAL_REGISTRY_URL","value":"https://harbor-nginx-lagoon-master.ch.amazee.io","scope":"internal_container_registry"},{"name":"INTERNAL_REGISTRY_USERNAME","value":"robot$drupal-example-test1-controller-1607911253","scope":"internal_container_registry"},{"name":"INTERNAL_REGISTRY_PASSWORD","value":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTExMTEyNTMsImlhdCI6MTYwNzkxMTI1MywiaXNzIjoiaGFyYm9yLXRva2VuLWRlZmF1bHRJc3N1ZXIiLCJpZCI6MzIxNTc5LCJwaWQiOjExNDMsImFjY2VzcyI6W3siUmVzb3VyY2UiOiIvcHJvamVjdC8xMTQzL3JlcG9zaXRvcnkiLCJBY3Rpb24iOiJwdXNoIiwiRWZmZWN0IjoiIn1dfQ.pn7suCNPTa9wnhncRuqZ3dAHCOdktSYzPvj9LYslxEZmawnn1apIOUdgFTvtaY5p4plfSBS4ZVgvo5SKGXBH9jkIQiZ1YVLyvvO9N7adNV-ljCSisZH3FhifdtPII2JLmQ5zBWN9HRR03rfJ2GBx5gvGtS-ZRElHYEEJZygSUto0i731OZhfHeeE15Qo7pHvChqZHaMrl9-0jfF7MQM9m4cw0ciZYTkCa2LEBKzieoOXg7r0yYRwfExiFMbvn0Ie5waqfdE9PFSGaXY4fpRH_D7vVE-wA7qf0f2ZKHKk4pf9vZPwRW-dPSKuDitnXrp-jgusGTzPHSs1Dt9Kd8qoWYa3YluUFuBrwTx2gtWUoh56tbn9G0ihLnrmL2eCTS4AG3VmmdQdaPczY6Ez8FuXzIoTTxfILcNSGXic3M27XF_dN7gt9jPi5_XBLzCVaTP4vzL3xyPrTuyuclyaUGBd6_FzTHydp5yIs5QKg3DHBJobzPxg3BdBjm3W6gynXvncZue9Rej2gGoyMkLOOmPKeVrxW6mnZjmwixnTupNoW06VKjzOYEdGzy0VbJBpcVM7clBCPXoltLIFCxlEeboysWSXTi8vM3ytxXA_zydxnn45DOtUBLAtCmRQVIrnaJL41iHkpgCmUOrYms4YdIda1P3mtaGxRsJDrB8G5dfAgzU","scope":"internal_container_registry"}]
- name: MONITORING_STATUSPAGEID
value: 'null'
resources: {}
volumeMounts:
- name: lagoon-deployer-token-r5b7f
readOnly: true
mountPath: /var/run/secrets/lagoon/deployer
- name: lagoon-sshkey
readOnly: true
mountPath: /var/run/secrets/lagoon/ssh
- name: default-token-mgnb7
readOnly: true
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Never
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: default
serviceAccount: default
nodeName: ip-10-200-123-50.eu-central-1.compute.internal
securityContext: {}
schedulerName: default-scheduler
tolerations:
- key: lagoon/build
operator: Exists
effect: NoSchedule
- key: lagoon/build
operator: Exists
effect: PreferNoSchedule
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 300
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 300
priorityClassName: lagoon-priority-production
priority: 1000000
enableServiceLinks: true
status:
phase: Failed
conditions:
- type: Initialized
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-12-24T15:11:27Z'
- type: Ready
status: 'False'
lastProbeTime: null
lastTransitionTime: '2020-12-24T15:11:27Z'
reason: ContainersNotReady
message: 'containers with unready status: [lagoon-build]'
- type: ContainersReady
status: 'False'
lastProbeTime: null
lastTransitionTime: '2020-12-24T15:11:27Z'
reason: ContainersNotReady
message: 'containers with unready status: [lagoon-build]'
- type: PodScheduled
status: 'True'
lastProbeTime: null
lastTransitionTime: '2020-12-24T15:11:27Z'
hostIP: 10.200.123.50
podIP: 10.200.87.81
podIPs:
- ip: 10.200.87.81
startTime: '2020-12-24T15:11:27Z'
containerStatuses:
- name: lagoon-build
state:
terminated:
exitCode: 128
reason: ContainerCannotRun
message: >-
OCI runtime create failed: container_linux.go:370: starting
container process caused: process_linux.go:459: container init
caused: setenv: invalid argument: unknown
startedAt: '2020-12-24T15:11:40Z'
finishedAt: '2020-12-24T15:11:40Z'
containerID: >-
docker://6b760f4c4ac254f15a031bbc94a420a953e503ea1e391ec5338494127b13c2de
lastState: {}
ready: false
restartCount: 0
image: 'imagecache.amazeeio.cloud/amazeeio/kubectl-build-deploy-dind:v1.13.1'
imageID: >-
docker-pullable://imagecache.amazeeio.cloud/amazeeio/kubectl-build-deploy-dind@sha256:3fdb6515cb513971adebbb2a1fb87cae860cdd47b34a0fd0f58f9d34f2e5cd86
containerID: >-
docker://6b760f4c4ac254f15a031bbc94a420a953e503ea1e391ec5338494127b13c2de
started: false
qosClass: BestEffort
with
- name: PR_NUMBER
value: "\0"
which then causes kubernetes to fail:
Error: failed to start container "lagoon-build": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: setenv: invalid argument: unknown
while I think this is actually a kubernetes bug that you should be able to create a pod with \0 as env variable, it's also a bug that the pull request number gets somehow swallowed by the controller.
I found the code that takes the message and creaetes the LagoonBuilds object:
https://github.com/amazeeio/lagoon-kbd/blob/main/handlers/message_queue.go#L95-L106 and I don't see anything wrong with it.
The only piece that I could imagine is that somehow the fact the value in the rabbitmq message is a string, but the lagoon types for LagoonBuilds is defined as a number:
https://github.com/amazeeio/lagoon-kbd/blob/86935f85c0516ee50deff1839000f7567f6b1b53/api/v1alpha1/lagoonbuild_types.go#L142
could cause some issues?
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent