Jenkins Create_Environment job failes with TLS Error about adop-docker-compose

[davidefreeman]

In order to remove SSL/TLS as the root caused, I disabled TLS by unsetting 'DOCKER_TLS_VERIFY' in the Jenkins build script. Unfortunately, Docker continued to fail, but at least without the TLS error. The error this time around is "Couldn't connect to Docker daemon - you might need to run docker-machine start default." when running 'docker-compose'.

I don't know how to make sure that docker daemon is listening on unecrypted port 4375, which is why I believe this error is appearing - docker is probably configured only for encrypted connections.

'''
[EnvInject] - Inject global passwords.
Started by user Admin User
[EnvInject] - Loading node environment variables.
[EnvInject] - Preparing an environment for the build.
[EnvInject] - Keeping Jenkins system variables.
[EnvInject] - Keeping Jenkins build variables.
[EnvInject] - Injecting as environment variables the properties content
WORKSPACE_NAME=ExampleWorkspace
PROJECT_NAME=ExampleWorkspace/ExampleProject

[EnvInject] - Variables injected successfully.
[EnvInject] - Injecting contributions.
Building remotely on Swarm_Slave-6839cf1d (swarm java8 ldap aws docker) in workspace /workspace/ExampleWorkspace/ExampleProject/Create_Environment

Deleting project workspace... done

[ssh-agent] Using credentials jenkins (ADOP Jenkins Master)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent] Java/JNR ssh-agent
[ssh-agent] Started.
Cloning the remote Git repository
Cloning repository ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template

git init /workspace/ExampleWorkspace/ExampleProject/Create_Environment # timeout=10
Fetching upstream changes from ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template
git --version # timeout=10
using GIT_SSH to set credentials ADOP Jenkins Master
git -c core.askpass=true fetch --tags --progress ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template +refs/heads/:refs/remotes/origin/
git config remote.origin.url ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template # timeout=10
Fetching upstream changes from ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template
using GIT_SSH to set credentials ADOP Jenkins Master
git -c core.askpass=true fetch --tags --progress ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template +refs/heads/:refs/remotes/origin/
git rev-parse refs/remotes/origin/master^{commit} # timeout=10
git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 86f2e0b110d7569d7933d77c6478d7e8e425c483 (refs/remotes/origin/master)
git config core.sparsecheckout # timeout=10
git checkout -f 86f2e0b110d7569d7933d77c6478d7e8e425c483
git rev-list 86f2e0b110d7569d7933d77c6478d7e8e425c483 # timeout=10
[Create_Environment] $ /bin/sh -xe /tmp/hudson6689850079724164446.sh

• set -x
• unset DOCKER_TLS_VERIFY
• env
  BUILD_URL=http://52.91.218.227/jenkins/job/ExampleWorkspace/job/ExampleProject/job/Create_Environment/11/
  JAVA_TARBALL=server-jre-8u45-linux-x64.tar.gz
  HOSTNAME=1b861298ccd5
  SWARM_MASTER=http://jenkins:8080/jenkins/
  HUDSON_SERVER_COOKIE=65676e8ba7afc02b
  WORKSPACE_NAME=ExampleWorkspace
  DOCKER_HOST=tcp://52.91.218.227:2376
  BUILD_TAG=jenkins-ExampleWorkspace-ExampleProject-Create_Environment-11
  DOCKER_NETWORK_NAME=local_network
  GIT_PREVIOUS_COMMIT=86f2e0b110d7569d7933d77c6478d7e8e425c483
  ROOT_BUILD_CAUSE=MANUALTRIGGER
  ENVIRONMENT_TYPE=DEV
  WORKSPACE=/workspace/ExampleWorkspace/ExampleProject/Create_Environment
  JOB_URL=http://52.91.218.227/jenkins/job/ExampleWorkspace/job/ExampleProject/job/Create_Environment/
  SLAVE_LABELS=aws ldap java8 docker
  SWARM_USER=jenkins
  SSH_AUTH_SOCK=/tmp/jenkins4448822328534665161.jnr
  GIT_AUTHOR_NAME=ADOP Jenkins
  GIT_COMMITTER_NAME=ADOP Jenkins
  SLAVE_EXECUTORS=1
  SLAVE_DESCRIPTION=Core Jenkins Slave
  NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
  GIT_COMMIT=86f2e0b110d7569d7933d77c6478d7e8e425c483
  SWARM_PASSWORD=ee42d31a287510567
  JENKINS_HOME=/var/jenkins_home
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  PROJECT_NAME=ExampleWorkspace/ExampleProject
  BUILD_CAUSE_MANUALTRIGGER=true
  GIT_COMMITTER_EMAIL=[email protected]
  INITIAL_ADMIN_PASSWORD=freeman87
  PWD=/workspace/ExampleWorkspace/ExampleProject/Create_Environment
  JAVA_HOME=/opt/java/jdk1.8.0_45
  HUDSON_URL=http://52.91.218.227/jenkins/
  JAVA_VERSION=1.8.0_45
  SLAVE_NAME=Swarm_Slave
  JOB_NAME=ExampleWorkspace/ExampleProject/Create_Environment
  XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
  BUILD_DISPLAY_NAME=#11
  BUILD_CAUSE=MANUALTRIGGER
  BUILD_ID=11
  JENKINS_URL=http://52.91.218.227/jenkins/
  HOME=/root
  DOCKER_CERT_PATH=//root/.docker/
  SHLVL=2
  INITIAL_ADMIN_USER=davidefreeman
  GIT_BRANCH=origin/master
  EXECUTOR_NUMBER=0
  JENKINS_SERVER_COOKIE=65676e8ba7afc02b
  GIT_URL=ssh://jenkins@gerrit:29418/ExampleWorkspace/ExampleProject/adop-cartridge-java-environment-template
  NODE_LABELS=Swarm_Slave-6839cf1d aws docker java8 ldap swarm
  HUDSON_HOME=/var/jenkins_home
  NODE_NAME=Swarm_Slave-6839cf1d
  BUILD_NUMBER=11
  ROOT_BUILD_CAUSE_MANUALTRIGGER=true
  HUDSON_COOKIE=da78031f-a79c-4bbc-b697-155309409c2b
  GIT_AUTHOR_EMAIL=[email protected]
  SLAVE_MODE=exclusive
  _=/usr/bin/env
• '[' DEV == DEV ']'
• createDockerContainer CI tomcat.conf
• echo CI, tomcat.conf
  CI, tomcat.conf
• export ENVIRONMENT_NAME=CI
• ENVIRONMENT_NAME=CI
  ++ echo ExampleWorkspace/ExampleProject
  ++ tr / _
• export SERVICE_NAME=ExampleWorkspace_ExampleProject_CI
• SERVICE_NAME=ExampleWorkspace_ExampleProject_CI
• docker-compose -p ExampleWorkspace_ExampleProject_CI up -d
  Couldn't connect to Docker daemon - you might need to run docker-machine start default.
  Build step 'Execute shell' marked build as failure
  [ssh-agent] Stopped.
  Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered
  Finished: FAILURE
  '''

[nickdgriffin]

Assuming you used "startup.sh" or "quickstart.sh" then Docker Machine will have created a TLS-enabled Engine, so trying to use it without TLS is not going to work without configuring it by hand (which isn't worth the effort really).

If you are running the scripts from Windows then Git Bash will have needed to be run as administrator and in Mac the script run with root access so that the certificates can be generated - otherwise it will silently fail and the Slave won't have been provided a certificate that allows this job to work. If you are launching an AWS machine which has been restarted then the IP may have changed which will require the certificates to first be fixed by Docker Machine (you can tell if you run "docker-machine ls" as it'll state it has timed out) and secondly to run "generate_client_certs.sh" to upload a new certificate to the Slave.

[davidefreeman]

That would be it - I did this install from my local Windows machine machine instead of my AWS Linux box. Rerunning quickstart now.

[davidefreeman]

Rerunning quickstart from git-bash with Admin privileges seems to have done the trick. Now I have to reread the documentation to see if I missed that line... :-)