ros-infrastructure / buildfarm_deployment_config

Example configuration for buildfarm_deployment repo

buildfarm_deployment_config

This is the repository with the example configurations for the buildfarm_deployment implementation. Please see the documentation in that repository for how to setup a buildfarm.

This is the example configs used for development and deployment on http://build.ros.org

buildfarm_deployment_config's Issues

reconfigure.bash uses 'xenial' branch of buildfarm_deployment?

Not sure whether this is actually an issue, but just to make sure I'm reporting this: reconfigure.bash on master seems to be checking for (and switching to in case it finds another branch has been checked out) the xenial branch of buildfarm_deployment (here).

Shouldn't this be master, now that the xenial migration is complete?

Adding import_indigo to reprepro config?

Would it be a good idea to add configuration file for importing upstream ros-indigo (or others) packages to https://github.com/ros-infrastructure/buildfarm_deployment_config/blob/master/repo/common.yaml#L132?

This way one would not need to manually perform this step: https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/custom_rosdistro.rst#import-ros-packages-which-are-not-built-by-the-custom-build-farm.

Example yaml file for all ros-indigo packages

name: backfill-ros
method: http://packages.ros.org/ros/ubuntu/
suites: [trusty]
component: main
architectures: [amd64, source]
filter_formula: Package (% ros-indigo-* )

Configure a separate repository component for debug packages

Per offline discussion continued from ros-infrastructure/bloom#32 (comment) we should set up a separate repository component to store debug packages in to avoid users installing them unintentionally with apt install 'ros-kinetic-*' or similar.

cc @tfoote @mikaelarguedas

Incorrect exit code running reconfigure.bash

$ sudo ./reconfigure.bash master
...
[Librarian] Copying .tmp/librarian/cache/source/git/b8ec0a48627bd713 to modules/jenkins
Running puppet
$ echo $?
1

openni conflicts on install

It appears to be a reappearance of this: ros/rosdistro#3986

Missing apt-transport-https

I am running Ubuntu Server 16.04 and encountered an issue with deploying the buildfarm using install_prerequisites.bash and reconfigure.bash.

It only deploys successfully when installing apt-transport-https first, before deploying with install_prerequisites.bash and reconfigure.bash. Otherwise it fails on installing Docker.

The running build.py fails

Running python build.py in a vanilla clone of this repo fails with the following error:

[Librarian] Installing tracywebtech-pip/1.3.2 
[Librarian] Executing puppet module install for tracywebtech-pip 1.3.2: puppet module install --version 1.3.2 --target-dir /root/buildfarm_deployment/master/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/tracywebtech-pip/1.3.2 --module_repository https://forgeapi.puppetlabs.com --modulepath /root/buildfarm_deployment/master/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/tracywebtech-pip/1.3.2 --module_working_dir /root/buildfarm_deployment/master/.tmp/librarian/cache/source/puppet/forge/forgeapi_puppetlabs_com/tracywebtech-pip/1.3.2 --ignore-dependencies tracywebtech-pip
Running puppet
The command '/bin/sh -c ./reconfigure.bash master' returned a non-zero code: 1
Traceback (most recent call last):
  File "build.py", line 37, in 
    subprocess.check_call(cmd.split())
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'master', '.']' returned non-zero exit status 1

Problem with connecting agents to master

I am trying to launch build farm using master branch

I have set all my configuration on common.yaml
and updated the master.yaml as well.

after running sudo bash ./reconfigure.bash for master, repo and agent.

Everything looks setup correctly, but the Jenkins agent fails to connect to the master.

The error is:

Jan 08, 2019 5:37:07 PM org.apache.commons.httpclient.auth.AuthChallengeProcessor selectAuthScheme
INFO: basic authentication scheme selected
Jan 08, 2019 5:37:07 PM org.apache.commons.httpclient.HttpMethodDirector processWWWAuthChallenge
INFO: Failure authenticating with BASIC 'Jenkins'@master:8080
Failed to fetch slave info from Jenkins CODE: 401
Retrying in 10 seconds

From further checking, On Jenkins Master, TCP port for JNLP agents is disables and also once enabling it, I must set Java Web Start Agent Protocol/2 which is deprecated and unencrypted.

Is there any proper workaround or fix?

Should public SSH key used by 'jenkins-agent' user be added to 'ssh_keys' section in 'common.yaml'?

Currently only the public key for the root user is added in the ssh_keys section of common.yaml. This allows passwordless logins over SSH to all hosts in a buildfarm (master, repo and slaves).

The comment above it is a bit cryptic:

The example public key should be changed but the private key has been deliberately deleted rather than leaked.

The trusty version of this had a second stanza in there for the public key the jenkins-slave user used to connect to the repo host (here).

Does the comment try to convey the fact that the public SSH key for the jenkins-agent user (to be used for logging into the repo host) should be added to the ssh_keys section there as well?

Or should it be added to the repo.yaml file, as this key should be imported on the repo host during deployment?

Checking for successful apt pkg installation?

Just ran into another case of #36 (ie: missing apt-transport-https, causing problems installing docker-ce later on), even though install_prerequisites.bash had been run.

Not sure how that happened, but would it make sense to check for presence of requirements after they've been installed (perhaps with something like dpkg -s <pkg_name>)?

I'm a fan of fail-early, so I'd rather have install_prerequisites.bash complain to me than seeing things run to completion only to see errors in puppet.log.

fix buildfarm deployment branch name

buildfarm_deployment_config/reconfigure.bash

Line 14 in 85c9118

BUILDFARM_DEPLOYMENT_BRANCH=jenkins-lts-upgrades

I'm trying to set a new clean instance of the buildfarm, but the process chashes because jenkins-lts-upgrades branch doesn't exist.

It seems to me that the correct branch is jenkins-lts-upgrade without s in the end.