Coder Social home page Coder Social logo

ci's Introduction

ROS 2 CI Infrastructure

This repository contains all of the scripts and resources for running batch CI jobs of ROS 2 repositories on a Jenkins build farm.

Setting up on a Jenkins build farm

You can run the create_jenkins_job.py Python3 script to create the jobs on your Jenkins build farm. This assumes you have a Jenkins build farm with at least one Linux agent, OS X agent, and Windows agent. However, even if you do not have the agents setup yet, you can create the jobs.

Before you run the script to create the jobs, you'll need to setup your environment according to this document:

https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst

This document will tell you what dependencies to install/setup like jenkinsapi and ros_buildfarm, as well as tell you how to setup authentication for Jenkins so that the script can login into your Jenkins build farm.

Once you've setup your environment, you can do a dry run with the following:

$ python3 create_jenkins_job.py --jenkins-url http://jenkins.example.com

The above will contact the Jenkins master at the provided url and try to authenticate. If successful, it will output the changes that would be made, but not commit them.

If the changes look good, then run the same command with the --commit option to actually make the changes:

$ python3 create_jenkins_job.py --jenkins-url http://jenkins.example.com --commit

You can also change the default location from which these CI scripts are pulled with options --ci-scripts-repository and --ci-scripts-default-branch. They allow you to change the default location from which to get the CI scripts, which is useful if you have forked ros2/ci. The defaults are [email protected]:ros2/ci.git and master respectively. The branch can be changed when running the job (it's a job parameter) to make it easy to test changes to the CI scripts using a branch on the main repository.

Adjusting batch CI jobs

The jobs will be mostly setup, but you may need to adjust some settings for your farm.

First you'll need to look at each job configuration and make sure that the 'Label Expression' for the 'Restrict where this project can be run' option matches some build agent label or build agent name for the operating system. For example, the default value for this in the Windows job is 'windows' because we (OSRF) run multiple Jenkins agents all with this label. You may want your job to run on a specific Windows agent in which case you can update the label to match the agent name/label you want to use.

Another thing to check is the credentials settings. The RTI binaries are provided by a git submodule which points to a private GitHub repository. You'll need to setup at least one ssh key in the Jenkins Credentials plugin and add the public part of that key as a deploy key on the private GitHub repository, currently https://github.com/osrf/rticonnextdds-src. Then on each job you'll want to make sure that this key is selected in two places. First under the 'Source Code Management' section, there is a 'Credentials' drop down box under the 'Repositories' section. Select the appropriate ssh key from that list. Second, under the 'SSH Agent' option, select the same ssh key from the drop down list called 'Credentials'. Note this option will not be there for Windows because I could not get the ssh-agent to work on Windows correctly and it is not needed anyways.

Using the batch CI jobs

Each of the batch CI jobs have the same set of parameters. The parameters have their own descriptions, but the main one to look at is the CI_BRANCH_TO_TEST parameter. It allows you to select a branch name across all of the repositories in the .repos file that should be tested. Repositories which have this branch will be switched to it, others will be left on the default branch, usually rolling.

Notes about MacOS, Windows Agents

Configuration details for running the Jenkins agent process via the Jenkins Swarm Client can be found in the configuration branch of this repository.

ci's People

Contributors

ahcorde avatar bhatsach avatar brawner avatar cclauss avatar christophebedard avatar clalancette avatar claraberendsen avatar cottsay avatar dhood avatar dirk-thomas avatar emersonknapp avatar esteve avatar gbiggs avatar gerkey avatar ivanpauno avatar j-rivero avatar jacobperron avatar jacquelinekay avatar karsten1987 avatar marcoag avatar mikaelarguedas avatar mjcarroll avatar nuclearsandwich avatar rohitsalem avatar rotu avatar ruffsl avatar sloretz avatar tfoote avatar thomas-moulard avatar wjwwood avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ci's Issues

Coverage builds are broken

When using --packages-up-to as build arguments on ci_linux_coverage, the following happens:

17:58:00 Package '--pytest-with-coverage' specified with --packages-up-to was not found

See for example: https://ci.ros2.org/job/ci_linux_coverage/20/.


nightly_linux_coverage is failing with a related error:

05:24:12 CMake Error: The source directory "/home/jenkins-agent/workspace/nightly_linux_coverage/ws/build/fastcdr/--pytest-with-coverage" does not exist.

Taken from https://ci.ros2.org/view/nightly/job/nightly_linux_coverage/736/console.

It has been failing since 01-15.

Retry nightlies on a different node if jobs fail because of setup issues

Sometimes nightly jobs fail because of setup issues, such as issues with Docker (e.g. nightly_linux_repeated 448) or Jenkins (e.g. nightly osx release 241). Currently we will notice these failures in the morning, take the failing nodes offline, and (maybe) re-run the jobs on another node during the day.

It would be handy if when jobs fail because of setup issues, the job is retried on another node automatically. In the case of the nightlies, it will save us from having to use the remaining online nodes for running nightly jobs during the daytime when we would probably prefer to use them for CI.

pip install fails with missing SSL module for Windows debug jobs

dirk-thomas/vcstool#42 was released to fix a regression and did so for all but one nightly job.

The Windows Debug nightly is still failing and after looking at why, it appears that the Python packages used during the jobs have not been updating since the switch to the debug interpreter was made.

13:14:18 Could not fetch URL https://pypi.python.org/simple/vcstool/: There was a problem confirming the ssl certificate: Can't connect to HTTPS URL because the SSL module is not available. - skipping
13:14:18 Requirement already up-to-date: vcstool in c:\python35\lib\site-packages

I didn't find anything that would easily let me enable or build ssl for the debug interpreter and when I attempted to bypass https by installing with the command

C:\> "C:\Python35\python_d.exe" -m pip install --index-url="http://pypi.python.org/simple" --trusted-host pypi.python.org -U EmPy coverage flake8 flake8-import-order mock nose pep8 pydocstyle pyflakes vcstool

Python still complained about HTTPS errors. I have temporarily brought the build back online by downloading the source archive for vcstool 0.1.26 and installing it with

C:\>"C:\Python35\python_d.exe" -m pip install -U --no-index --find-links="file://%USERPROFILE%/Downloads/vcstool-0.1.26.tar.gz" vcstool

This started with build http://ci.ros2.org/view/nightly/job/nightly_win_deb/469/ Introduced by #72

Potential permanent fixes:

  • Find out how to successfully bypass https when accessing pypi using the Debug interpreter
  • Build or install the ssl module for the debug interpreter
  • Write something that checks the pypi index and source downloads the latest versions of our packages using the standard python interpreter then passes them to the debug interpreter to be installed.

As of now I've only updated vcstool and only on icecube. Now that that's building I'm going to run around and update the other windows hosts in similar fashion.

Missing ros-dashing-ros2action in ros-dashing-desktop deb package

Not sure if this is the correct repo for this issue, if not, please point me in the right direction.

When installing ros-dashing-desktop through the DEB packages (sudo apt install ros-dashing-desktop) it seems to leave out the cli package for actions. You have to do an additional sudo apt install ros-dashing-ros2action.

Not sure where this is configured, if someone wants to point me in the right direction I can try to fix and create a PR.

Thanks

Thread sanitizer job frequently fails with 'java.lang.OutOfMemoryError: Java heap space'

Memory usage is high, higher than it should be (JVM is allotted a maximum heap size of 10G on a 16G instance) on the CI host but I am still verifying whether the OutOfMemoryError is on the Jenkins master rather than the agent JVM process.

Introduced in #254 and is blocking ros2/ros2#666 👿

Jenkins stacktrace
ERROR: Step ‘Publish xUnit test result report’ aborted due to exception:
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1181)
        at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1134)
        at com.sun.org.apache.xerces.internal.dom.DeferredTextImpl.synchronizeData(DeferredTextImpl.java:96)
        at com.sun.org.apache.xerces.internal.dom.CharacterDataImpl.getNodeValue(CharacterDataImpl.java:91)
        at net.sf.saxon.dom.NodeWrapper.getStringValueCS(NodeWrapper.java:364)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:587)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:570)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:570)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:570)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:570)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:570)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.om.Navigator.copy(Navigator.java:570)
        at net.sf.saxon.dom.NodeWrapper.copy(NodeWrapper.java:848)
        at net.sf.saxon.event.DocumentSender.send(DocumentSender.java:54)
        at net.sf.saxon.event.Sender.sendDocumentInfo(Sender.java:278)
        at net.sf.saxon.event.Sender.send(Sender.java:177)
        at net.sf.saxon.event.Sender.send(Sender.java:50)
        at net.sf.saxon.Configuration.buildDocument(Configuration.java:2973)
        at net.sf.saxon.s9api.DocumentBuilder.build(DocumentBuilder.java:287)
        at org.jenkinsci.lib.dtkit.util.converter.ConversionService.convert(ConversionService.java:331)
        at org.jenkinsci.lib.dtkit.util.converter.ConversionService.convert(ConversionService.java:172)
        at org.jenkinsci.lib.dtkit.model.InputMetricXSL.convert(InputMetricXSL.java:181)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-172-30-0-91.ec2.internal/172.30.0.91:59622 failed
        at hudson.remoting.Channel.call(Channel.java:961)
        at hudson.FilePath.act(FilePath.java:1072)
        at hudson.FilePath.act(FilePath.java:1061)
        at org.jenkinsci.plugins.xunit.XUnitProcessor.performTests(XUnitProcessor.java:145)
        at org.jenkinsci.plugins.xunit.XUnitProcessor.performXUnit(XUnitProcessor.java:88)
        at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:142)
        at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:134)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

wrong directory path to COLCON_IGNORE marker file?

I've seen my test failing with the following error message:

16:29:32 FileNotFoundError: [Errno 2] No such file or directory: 'src/src/ros2/common_interfaces/actionlib_msgs/COLCON_IGNORE'

Printing the individual variables such as package_name and package_path reveal a duplicate src string within the path. I suppose that removing the first src would help that.

package_name, package_path, _ = line.split('\t', 2)

16:29:32 package_name action_msgs
16:29:32 package_path src/ros2/rcl_interfaces/action_msgs
16:29:32 args sourcespace src
16:29:32 package_name action_tutorials_cpp
16:29:32 package_path src/ros2/demos/action_tutorials/action_tutorials_cpp
....

any idea why this occurring right now?

Help for ros2 ci on mac platform

Hi all,

Now, we are developing a Node.js client named rclnodejs of ros2.0, and we want it can support Linux, MacOS and Windows platforms. But we encounter a problem that we can't find a suitable ci service on mac, actually we have tried travic-ci (it has a maximum 50min build limitation) and circleci (doesn't have a free plan on mac).

I am wondering if we can leverage the ci of http://ci.ros2.org/, or do you have some better suggestions? Your help is appreciated!

Minggang

Linux jobs failing to run vcs from venv if a system version is installed.

The batch script expands the vcs binary to its path in the venv and hard codes that path in the command.

vcs_cmd = [
'"%s"' % job.python,
'"%s"' % os.path.join(venv_path, 'bin', 'vcs')]

Recent Linux CI jobs have started omitting vcs from the venv since it is installed on the system already and provided by /usr/bin/vcs

From ci_linux #5955

==> "/home/rosbuild/ci_scripts/venv/bin/python" -m pip install -U EmPy coverage catkin_pkg flake8 flake8-blind-except flake8-builtins flake8-class-newline flake8-comprehensions flake8-deprecated flake8-docstrings flake8-import-order flake8-quotes git+https://github.com/lark-parser/lark.git@8415fa26a3d3d2b81e64e4fe440faab15b53db49 mock nose pep8 pydocstyle pyflakes pyparsing pytest pytest-cov pytest-repeat pytest-rerunfailures pytest-runner pyyaml vcstool colcon-core colcon-defaults colcon-library-path colcon-metadata colcon-output colcon-package-information colcon-package-selection colcon-parallel-executor colcon-powershell colcon-python-setup-py colcon-recursive-crawl colcon-test-result colcon-cmake colcon-ros colcon-bash colcon-zsh
... output elided ...
03:30:01 Collecting vcstool
03:30:01   Downloading https://files.pythonhosted.org/packages/9f/23/0839fbd992b5cc5f7dcb3751e9d8f82eabe5923212e76ddead457dd4a6b0/vcstool-0.1.37-py3-none-any.whl

From ci_linux #5956

==> "/home/rosbuild/ci_scripts/venv/bin/python" -m pip install -U EmPy coverage catkin_pkg flake8 flake8-blind-except flake8-builtins flake8-class-newline flake8-comprehensions flake8-deprecated flake8-docstrings flake8-import-order flake8-quotes git+https://github.com/lark-parser/lark.git@8415fa26a3d3d2b81e64e4fe440faab15b53db49 mock nose pep8 pydocstyle pyflakes pyparsing pytest pytest-cov pytest-repeat pytest-rerunfailures pytest-runner pyyaml vcstool colcon-core colcon-defaults colcon-library-path colcon-metadata colcon-output colcon-package-information colcon-package-selection colcon-parallel-executor colcon-powershell colcon-python-setup-py colcon-recursive-crawl colcon-test-result colcon-cmake colcon-ros colcon-bash colcon-zsh
... output elided ...
Requirement already up-to-date: vcstool in /usr/lib/python3/dist-packages (0.1.38)

But since the script always looks for /home/rosbuild/ci_scripts/venv/bin/vcs this is causing the job to fail.

==> "/home/rosbuild/ci_scripts/venv/bin/python" "/home/rosbuild/ci_scripts/venv/bin/vcs" import "src" --force --retry 5 --input 00-ros2.repos
/home/rosbuild/ci_scripts/venv/bin/python: can't open file '/home/rosbuild/ci_scripts/venv/bin/vcs': [Errno 2] No such file or directory
<== '. ../venv/bin/activate && "/home/rosbuild/ci_scripts/venv/bin/python" "/home/rosbuild/ci_scripts/venv/bin/vcs" import "src" --force --retry 5 --input 00-ros2.repos' exited with return code '2'

Quotes in build/test args are not always preserved

For the "repeated" tests, we pass the test argument --pytest-args -m "not linter".

On the Ubuntu workers, this is broken into the arguments: ['--pytest-args', '-m', '"not', 'linter"']
On CentOS, the double-quotes are dropped and we get: ['--pytest-args', '-m', 'not linter']

Even though the quoted argument is broken apart on Ubuntu, the double-quotes are preserved and handed through to the invocation of colcon.

On CentOS, there are no quotes and colcon sees --pytest-args -m not linter, and pytest therefore sees -m not linter which results in failures like:

ERROR: file not found: linter

This is the cause of the failing nightly_linux-centos_repeated jobs.

compile options for ros2 builds

@nuclearsandwich @dirk-thomas

I've noticed in one of your build job outputs for ros2: http://build.ros2.org/view/Abin_uX64/job/Abin_uX64__rclcpp__ubuntu_xenial_amd64__binary/11/consoleText that you use the following compile options:
-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wpedantic.

We recently also came across this article: https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/ and noticed in particular these flags:

  1. -fstack-protector-strong
  2. -Werror=format-security
  3. -D_FORTIFY_SOURCE=2

Q1: where do you set these above flags because I could not really spot them in any CMakeLists files in any of the major repos on https://github.com/ros2.

Q2: are you aware that any of above options introduces a significant runtime performance overhead?

Install pytest, setuptools from apt on Bionic

The Linux Development Setup instructions currently say to install pytest and setuptools from pip on xenial. It sounds like they could be installed via apt instead.

# [Ubuntu 16.04] install extra packages not available or recent enough on Xenial
python3 -m pip install -U \
  pytest \
  pytest-cov \
  pytest-runner \
  setuptools

However, CI currently installs all of these from pip. This issue is to change CI to install these from apt and see if it works.

See also ros2/ros2_documentation#67
If the dependencies from apt work then this PR will need to be undone: ros2/ros2_documentation#70

No access to "build farmer logbook"

ci.ros2.org links to the "build farmer logbook". However, clicking on that link only yields a "permission denied" response.

If that thing is internal, no problem, but I would recommend removing the link, or adding a note that its internal.

If its not supposed to be internal, please have a look :-)

Turtlebot CI job

Here's what I have in mind for creating a CI job for the turtlebot demo. I am looking to get the direction approved before speaking with @nuclearsandwich about how to implement it.

As I imagine it, this will be a manually-triggered CI job while we're developing the demo and then can switch to a nightly once it's stable.

  1. Add the supplementary repos file as a jenkins parameter
    1. Modify the ci_job.xml template to include this parameter if something like supplementary_repos_file is set when evaluating the template.
  2. Add the ci_linux-turtlebot_demo job to the farm
    1. Add it to the create_jenkins_job.py script
    2. Set the default supplementary repos file to https://raw.githubusercontent.com/ros2/turtlebot2_demo/master/turtlebot2_demo.repos or something (using a similar strategy to 38f57f3#diff-10b89ee3f1ac5a693c0ff698ad83c16dR1)
  3. Pull the supplementary repos file into the source space
    1. Put a for-loop around this section that calls vcs
  4. For turtlebot demo dependencies available as binaries, only add them to the dockerfile if necessary
    1. Inject the installation of the dependencies into the dockerfile using sed as is done to modify the FROM command for arm jobs -- OR --
    2. Use the create_dockerfile method from ros_buildfarm to generate the Dockerfile from a template, only including turtlebot dependency snippet if appropriate
  5. Ask turtlebot demo developers to target PRs at the turtlebot2_demo.repos file, and PRs for additional binary dependencies at the turtlebot demo extra dependencies list on ros2/ci

I understand point 4 as just a "nice to have".

Docker containers on the same buildfarm host interfering with each other

Our tests that check for nodes with particular names occasionally detect "unexpected" nodes.

It usually looks something like (output from this ARM repeated nightly):

03:30:27 9: /home/rosbuild/ci_scripts/ws/src/ros2/rcl/rcl/test/rcl/test_get_node_names.cpp:91: Failure
03:30:27 9: Expected equality of these values:
03:30:27 9:   size_t(2)
03:30:27 9:     Which is: 2
03:30:27 9:   node_names.size
03:30:27 9:     Which is: 3
03:30:27 9: [test_rcl_get_node_names]: Found node names:
03:30:27 9: node1
03:30:27 9: ros1_bridge_test_client
03:30:27 9: node2

I'd been thinking it was from zombie nodes from the same machine, that somehow didn't get cleaned up properly when tests timed out or something. However, ARM repeated nightlies don't even test the bridge, which is what tipped me off to the possibility of the interference coming from a separate build. Indeed this packaging build was running bridge tests at 3:30 PST that day.

Unlike the OS X/Windows nodes which have custom ROS_DOMAIN_IDs to avoid interference, we run our linux builds containerised and don't set custom ROS_DOMAIN_IDs. This wasn't an issue until we swapped our ARM linux jobs to a single native ARM host that has multiple executors that can run in parallel (as opposed to our AWS x86 linux hosts that run parallel jobs on different machines). By default the containers will start on the default Docker bridge network that connects them all, which is why they are interfering with each other's tests.

For cross-reference, there was a time where OS X nodes were cross-talking like this and it came down to the domain ID too: ros2/build_farmer#89

Possible solutions:

  1. Set a custom ROS_DOMAIN_ID for the builds. Ideally unique to the host:executor combo, but even a randomised ID would reduce the chance of interference.
  2. Create custom docker networks for each executor.
  3. Perhaps it's possible to modify the default network to prevent inter-container communication without modifying the domain ID.
  4. Surely many other networking configurations that I haven't thought of 😄

Most straightforward thing (because I am not familiar with how to access the ID of the executor on which the build is running) seems like injecting a random ID into the Dockerfile like we do with the date; input on the direction welcome.

7zip unpack fails

Under windows while unpacking the binary package with 7zip a strange behaviour appears. Windows is not able to read certain files properly any more. More details are described in this issue:

ros2/rviz#208

There is also a suggestion mentioned what causes this issue and could fix it.

@greimela-si:

The real cause seems to be the X file attribute.

Reading the file fails if the X attribute is set and the A attribute is not set.
This is why I originally thought that the A attribute might be responsible.

Add/change turtlebot demo to build on aarch64

The current turtlebot_demo CI job runs on labels marked as "linux", which in our build farm is implicitly x86_64 machines. However, the real turtlebot demo is almost certainly going to run on aarch64, so we should either change the label on the turtlebot-demo job to be "linux_aarch64", or we should add an additional job to build turtlebot-demo on both architectures. @nuclearsandwich @gerkey does that make sense?

inconsistent test count reported in Jenkins if tests are flaky

The nightly jobs reruns failed test with ctest --rerun-failed. But since this run overwrites the generated Test.xml Jenkins will report a different number of test because tests which passed in the initial run are not included in the new result file anymore. I don't think there is much we can do (except parsing the result file before and after and implementing our own merge strategy). I have reported the problem upstream, maybe it gets addressed in CTest directly: https://gitlab.kitware.com/cmake/cmake/issues/16314

libasio-dev recommended packages installed?

I noticed in http://ci.ros2.org/job/ci_linux/2476/consoleFull (step 32) that all the recommended packages for libasio-dev were installed. Is this the expected behavior?

apt command ran:

Step 32 : RUN apt-get update && apt-get install -y libasio-dev libtinyxml2-dev valgrind libssl-dev

resulting installed packages:

  gdb gdbserver icu-devtools libbabeltrace-ctf1 libbabeltrace1
  libboost-date-time-dev libboost-date-time1.58-dev libboost-date-time1.58.0
  libboost-dev libboost-regex-dev libboost-regex1.58-dev libboost-regex1.58.0
  libboost-serialization1.58-dev libboost-serialization1.58.0 libboost1.58-dev
  libc6-dbg libicu-dev libssl-doc zlib1g-dev

AFAICT all these boost packages are recommended by libasio-dev but not required by any package.

Transition Windows/Mac CI farms to cloud virtualization

This issue tracks the progress of transitioning the Windows CI/Mac CI machines to cloud virtualization. The goal of the project is to create a more reproducible, deployable and scalable Jenkins CI farm. Because Microsoft has embraced containerization in the last couple of years, it is feasible to run Windows CI instances inside docker containers running on VMs from a cloud provider. On the mac side, MacStadium offers potential for virtualization with mac instances.

Window cloud virtualization tasks:

  • Demonstrate a CI build by running run_ros2_batch.py from a docker container
  • Deploy a Jenkin's build job to an EC2 instance
    https://citest.ros2.org/job/ci_windows/87/
  • Merge windows_docker_resources PR (#361) (1/31/2019)
  • Cloud Jenkins agents running side-by-side with current CI farm (1/31/2019)
  • Windows configuration management of EC2 VMs with Chef (1/31/2019)
  • Retire bare metal CI Windows servers (3/1/2019)

Mac Cloud Virtualization tasks (TBD)

  • Investigate MacStadium's cloud virtualization offering
  • Demostrate CI build by running run_ros2_batch.py from virtual mac instance
  • Deploy a Jenkin's build job to a MacStadium instance
  • Create and Merge Mac CI PR
  • Cloud Jenkins agents running side-by-side with current CI farm
  • Mac configuration management of MacStadium VMs with Orka
  • Retire bare metal CI Mac servers

Daily invalidation of dockerfiles isn't working

We are using RUN echo "@today_str" in our Dockerfile to invalidate it daily, but unlike the dockerfiles in the ros1 buildfarm (e.g. this one), the dockerfile in the ros2 buildfarm is not an empy template.

As a result we just end up printing the string (from http://ci.ros2.org/job/ci_turtlebot-demo/3/consoleFull):

17:12:55 Step 40 : RUN echo "@today_str"
17:12:55  ---> Running in 7e19d30c09ba
17:12:55 @today_str

and then using the image from the cache on all successive builds (from http://ci.ros2.org/view/packaging/job/packaging_linux/611/consoleFull):

14:05:04 Step 39 : RUN echo "@today_str"
14:05:04  ---> Using cache
14:05:04  ---> 701834003880

Build C++ application failed with last successful package on Linux

Hi there,

Recently I noticed that there was a persistent link error when compiling a simple example written in C++.

/usr/bin/ld: warning: libcrypto.so.1.1, needed by /home/minggang/Downloads/ros2-linux/lib/libfastrtps.so.1.5.0, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libtinyxml2.so.6, needed by /home/minggang/Downloads/ros2-linux/lib/libfastrtps.so.1.5.0, not found (try using -rpath or -rpath-link)
/home/minggang/Downloads/ros2-linux/lib/libfastrtps.so.1.5.0: undefined reference to `BUF_MEM_free@OPENSSL_1_1_0'
/home/minggang/Downloads/ros2-linux/lib/libfastrtps.so.1.5.0: undefined reference to `tinyxml2::XMLNode::Value() const'
/home/minggang/Downloads/ros2-linux/lib/libfastrtps.so.1.5.0: undefined reference to `EVP_PKEY_new@OPENSSL_1_1_0'
/home/minggang/Downloads/ros2-linux/lib/libfastrtps.so.1.5.0: undefined reference to `X509_STORE_CTX_free@OPENSSL_1_1_0'

Detailed log see here

More information:

$ldd libfastrtps.so.1.5.0 
./libfastrtps.so.1.5.0: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by ./libfastrtps.so.1.5.0)
	linux-vdso.so.1 =>  (0x00007ffce69ba000)
	libfastcdr.so.1 => /home/minggang/Downloads/ros2-linux/lib/libfastcdr.so.1 (0x00007f9326b83000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9326966000)
	libcrypto.so.1.1 => not found
	libtinyxml2.so.6 => not found
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f93265e4000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f93263ce000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9326004000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9327222000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9325cfb000)

This problem can not be reproduced on macOS and Windows, is there any dependency change?

The latest packaging of Linux platform is built on ardent

Hi,

I noticed that the latest successful Linux build, build 970, is against ardent. Have the ci policy changed?

I want to download the latest packaging which is on master branch to verify our project rclnodejs and ros2-web-bridge, but we met the problem that the code of rcl was not on the master branch. Where can I get the master packaging?

Thanks!

What is the best place to add build argument to packaging job?

Hey guys, I'd like the Security feature to be part of the packaging jobs so that people can try it out from "binaries" on every platform. The packaging jobs don't take "ament build" parameters. What is the best thing to do here ? add a parameter also available from the UI? or add tham manually in the packaging job template?

Thamks!

end delimiter added when not needed

The logic for adding -- automatically at the end of the argument line seems to still be too simple for some edge cases.
I tried to pass --parallel --cmake-args -DSECURITY=ON -- --end-with rmw_fastrtps_cpp to CI and tried to run:
/usr/local/bin/python3 -u run_ros2_batch.py --do-venv --force-ansi-color --workspace-path /Users/osrf/jenkins/workspace/ci_osx --connext --disable-connext-dynamic --fastrtps --isolated --ament-build-args --parallel --cmake-args -DSECURITY=ON -- --end-with rmw_fastrtps_cpp -- --ament-test-args --retest-until-pass 10 --end-with rmw_fastrtps_cpp -- and failed with error: unrecognized arguments: --end-with rmw_fastrtps_cpp --

My conclusion is that greedy argument has to be the last otherwise triple hyphen should be used as defined in osrf_pycommon

Link to job with greedy arg in the middle: http://ci.ros2.org/job/ci_osx/1901/console#console-section-1
Link to job with greedy arg as last arg: http://ci.ros2.org/job/ci_osx/1902/consoleFull#console-section-1

Get builds going on packet.net

Notes to self:

root@ros2-testing:~/code/ci# history
    3  apt-get update
    4  apt-get install docker.io
    6  mkdir code
    7  cd code
    8  git clone https://github.com/ros2/ci
    9  cd ci
   12  git checkout packet_aarch64
   20  service docker start
   21  docker build --build-arg PLATFORM=arm -t ros2_batch_ci_aarch64 linux_docker_resources
   23  mkdir ~/.ccache
   26  docker run --privileged -e UID=1001 -e GID=1001 -e CI_ARGS="--do-venv --force-ansi-color --workspace-path /root/code/ci --fastrtps --isolated --cmake-build-type Debug --ament-build-args --parallel -- --ament-test-args --ctest-args -LE linter --" -e CCACHE_DIR=/home/rosbuild/.ccache -i -v /root/code/ci:/home/rosbuild/ci_scripts -v /root/.ccache:/home/rosbuild/.ccache ros2_batch_ci_aarch64

Fat archives not relocatable

The linux and osx packaging jobs are not currently generating re-locatable prefix.sh scripts.

Output on OS X:

$ source prefix.sh
not found: "/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install/share/ament_cppcheck/package.sh"
not found: "/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install/share/ament_flake8/package.sh"
not found: "/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install/share/ament_pep8/package.sh"
not found: "/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install/share/console_bridge/package.sh"
not found: "/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install/share/fastcdr/package.sh"
not found: "/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install/share/gtest_vendor/package.sh"
...

Contents:

# generated from colcon_core/shell/template/prefix.sh.em

# function to source another script with conditional trace output
# first argument: the name of the script file
colcon_prefix_source_shell_script() {
  # arguments
  _colcon_prefix_source_shell_script="$1"

  # source script with conditional trace output
  if [ -f "$_colcon_prefix_source_shell_script" ]; then
    if [ -n "$COLCON_TRACE" ]; then
      echo ". \"$_colcon_prefix_source_shell_script\""
    fi
    . "$_colcon_prefix_source_shell_script"
  else
    echo "not found: \"$_colcon_prefix_source_shell_script\"" 1>&2
  fi

  unset _colcon_prefix_source_shell_script
}


# since a plain shell script can't determine its own path when being sourced
# it uses the destination set at build time
COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/ament_cppcheck/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/ament_flake8/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/ament_pep8/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/console_bridge/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/fastcdr/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/gtest_vendor/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/orocos_kdl/package.sh"

COLCON_CURRENT_PREFIX=/Users/osrf/jenkins-agent/workspace/packaging_osx/ws/install
colcon_prefix_source_shell_script "$COLCON_CURRENT_PREFIX/share/osrf_pycommon/package.sh"

...

unset COLCON_CURRENT_PREFIX
unset colcon_prefix_source_shell_script

Contents for linux is similar but uses COLCON_CURRENT_PREFIX=/home/rosbuild/ci_scripts/ws/install

retest-until-fail vs repeat-until-fail

By investigating flaky tests I noticed that the repeated nightlies are using the --retest-until-fail <N> option, whereas when testing locally I run ament test with the option --ctest-args --repeat-until-fail <N> --

I'm wondering if --retest-until-fail <N> has the desired behavior for our repeated jobs. Below explained the difference of behavior between the two options

  • ctest --repeat-until-fail <N>:
    Ensures that all tests are run until failure or N runs
for test in tests_of_this_package:
    for i in range(N):
        rc = run_test(test)
        if rc:
           break  # move on to the next TEST
  • ament test --retest-until-fail <N>
    Stop testing a package as soon as a single test of this package fails
for i in range(N):
    rc = 0
    for test in tests_of_this_package:
        rc += run_test(test)
    if rc:
        break # move on to the next PACKAGE

This could explain why the reported flaky tests are different every night. Especially on packages with a lot of tests like the test_communication

Options we may want to consider:

  • change the default test argument for the repeated nightlies to use the ctest option repeat-until-fail
  • modify ament behavior to blacklist the failed tests but repeat the passed ones up to N times
  • do a run or two with the ctest --repeat-until-fail <N> argument to have an exhaustive list of the flaky tests

To have an idea of how long the nightlies would take with ctest --repeat-until-fail 20 I launched http://ci.ros2.org/job/ci_windows/2030/, on windows it took 5h30min instead of 2h

Stop building the tests in packaging jobs?

We build the tests of all packages in the packaging jobs but we don't test them.
Is there a drawback if we stop building the tests? (we would keep building the bridge and test it)

Shebang lines not re-written for executables in libexec

Our packaging jobs re-write shebang lines to be portable, however it only seems to do so for files installed to 'bin':

bin_path = os.path.join(args.installspace, 'bin')

Since we install executables to "libexec" as of late, this causes the packaged binaries to give this error (OS X):

$ ros2 run demo_nodes_py talker
Traceback (most recent call last):
  File "/Users/dhood/Downloads/ros2-osx-june25-2017/bin/ros2", line 11, in <module>
    load_entry_point('ros2cli==0.0.0', 'console_scripts', 'ros2')()
  File "/Users/dhood/Downloads/ros2-osx-june25-2017/lib/python3.6/site-packages/ros2cli/cli.py", line 64, in main
    rc = extension.main(parser=parser, args=args)
  File "/Users/dhood/Downloads/ros2-osx-june25-2017/lib/python3.6/site-packages/ros2run/command/run.py", line 59, in main
    return run_executable(path=path, argv=args.argv)
  File "/Users/dhood/Downloads/ros2-osx-june25-2017/lib/python3.6/site-packages/ros2run/api/__init__.py", line 75, in run_executable
    completed_process = subprocess.run([path] + argv)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/dhood/Downloads/ros2-osx-june25-2017/lib/demo_nodes_py/talker'

$ cat /Users/dhood/Downloads/ros2-osx-june25-2017/lib/demo_nodes_py/talker
#!/Users/osrf/jenkins/workspace/packaging_osx/venv/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'demo-nodes-py==0.0.0','console_scripts','talker'
__requires__ = 'demo-nodes-py==0.0.0'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('demo-nodes-py==0.0.0', 'console_scripts', 'talker')()
    )

False positive test passing scenario when setup.py has issues

See:

https://ci.ros2.org/job/ci_linux/4662/consoleFull#console-section-172

Basically I had a bug in my setup.py, which is fine, but the tests don't run and despite having an error message, end up resulting in a "SUCCESS" jenkins job.

Note I was using --packages-select launch_ros.

I don't have time to dig into my self, but I thought I put up a notice for others to double check their tests are actually running.

Here's the relevant output in case the jenkins job gets deleted or something:

6.10.1. launch_ros
Hide Details

Starting >>> launch_ros
00:45:40 running pytest
00:45:40 Searching for rclpy
00:45:40 Reading https://pypi.org/simple/rclpy/
00:45:40 Couldn't find index page for 'rclpy' (maybe misspelled?)
00:45:40 Scanning index of all packages (this may take a while)
00:45:40 Reading https://pypi.org/simple/
00:45:47 No local packages or working download links found for rclpy
00:45:47 error: Could not find suitable distribution for Requirement.parse('rclpy')
00:45:47 
6.10.2. launch_ros (stderr)
Hide Details

--- stderr: launch_ros
00:45:47 Couldn't find index page for 'rclpy' (maybe misspelled?)
00:45:47 No local packages or working download links found for rclpy
00:45:47 error: Could not find suitable distribution for Requirement.parse('rclpy')
00:45:47 ---
00:45:47 Finished <<< launch_ros [7.58s]	[ with test failures ]
00:45:47 
00:45:47 Summary: 1 package finished [8.38s]
00:45:47   1 package had stderr output: launch_ros
00:45:47   1 package had test failures: launch_ros
00:45:47 
00:45:47 II> colcon test returned: '0'
00:45:47 # END SUBSECTION
00:45:47 
6.11. test-result --all
Hide Details

# BEGIN SUBSECTION: test-result --all
00:45:47 (RTI)
00:45:47 (venv)
00:45:47 ==> . "/home/rosbuild/ci_scripts/ws/rti.sh" && /home/rosbuild/ci_scripts/venv/bin/colcon test-result --build-base "build" --all
00:45:47 ###############################################################################
00:45:47 
00:45:47          (c) Copyright, Real-Time Innovations, All rights reserved.            
00:45:47                                                                                
00:45:47                            RTI Connext DDS 5.3.1                       
00:45:47 
00:45:47 ###############################################################################
00:45:47 Summary: 0 tests, 0 errors, 0 failures, 0 skipped
00:45:47 
00:45:47 II> colcon test-result returned: '0'
00:45:47 # END SUBSECTION
00:45:47 
6.12. test-result
Hide Details

# BEGIN SUBSECTION: test-result
00:45:47 (RTI)
00:45:47 (venv)
00:45:47 ==> . "/home/rosbuild/ci_scripts/ws/rti.sh" && /home/rosbuild/ci_scripts/venv/bin/colcon test-result --build-base "build"
00:45:47 ###############################################################################
00:45:47 
00:45:47          (c) Copyright, Real-Time Innovations, All rights reserved.            
00:45:47                                                                                
00:45:47                            RTI Connext DDS 5.3.1                       
00:45:47 
00:45:47 ###############################################################################
00:45:48 Summary: 0 tests, 0 errors, 0 failures, 0 skipped
00:45:48 
00:45:48 II> colcon test-result returned: '0'
00:45:48 # END SUBSECTION
00:45:48 
6.13. ccache stats (after)
Hide Details

# BEGIN SUBSECTION: ccache stats (after)
00:45:48 (RTI)
00:45:48 (venv)
00:45:48 ==> . "/home/rosbuild/ci_scripts/ws/rti.sh" && ccache -s
00:45:48 ###############################################################################
00:45:48 
00:45:48          (c) Copyright, Real-Time Innovations, All rights reserved.            
00:45:48                                                                                
00:45:48                            RTI Connext DDS 5.3.1                       
00:45:48 
00:45:48 ###############################################################################
00:45:48 cache directory                     /home/rosbuild/.ccache
00:45:48 primary config                      /home/rosbuild/.ccache/ccache.conf
00:45:48 secondary config      (readonly)    /etc/ccache.conf
00:45:48 cache hit (direct)                377050
00:45:48 cache hit (preprocessed)          157942
00:45:48 cache miss                        216679
00:45:48 cache hit rate                     71.17 %
00:45:48 called for link                   233608
00:45:48 compile failed                       133
00:45:48 preprocessor error                   335
00:45:48 no input file                         71
00:45:48 cleanups performed                     0
00:45:48 files in cache                     18393
00:45:48 cache size                           4.5 GB
00:45:48 max cache size                       5.0 GB
00:45:48 
00:45:48 # END SUBSECTION
00:45:49 + echo # END SECTION
00:45:49 # END SECTION
00:45:49 $ ssh-agent -k
00:45:49 unset SSH_AUTH_SOCK;
00:45:49 unset SSH_AGENT_PID;
00:45:49 echo Agent pid 16587 killed;
00:45:49 [ssh-agent] Stopped.
00:45:49 [WARNINGS] Parsing warnings in console log with parser CMake
00:45:49 [WARNINGS] Found 0 warnings after exclusion.
00:45:49 <Git Blamer> Using GitBlamer to create author and commit information for all warnings.
00:45:49 <Git Blamer> GIT_COMMIT=f47f0ef8a6aae0faad63f7e93f71ca63e62c3599, workspace=/home/jenkins-agent/workspace/ci_linux
00:45:49 [WARNINGS] Parsing warnings in console log with parser GNU C Compiler 4 (gcc)
00:45:49 [WARNINGS] Found 0 warnings after exclusion.
00:45:49 <Git Blamer> Using GitBlamer to create author and commit information for all warnings.
00:45:49 <Git Blamer> GIT_COMMIT=f47f0ef8a6aae0faad63f7e93f71ca63e62c3599, workspace=/home/jenkins-agent/workspace/ci_linux
00:45:49 [WARNINGS] Computing warning deltas based on reference build #4658
00:45:49 [WARNINGS] Ignore new warnings since this is the first valid build
00:45:49 [WARNINGS] Plug-in Result: Success - no threshold has been exceeded
00:45:49 [Cobertura] Publishing Cobertura coverage report...
00:45:49 [Cobertura] No coverage results were found using the pattern 'ws/build*/**/*coverage.xml' relative to '/home/jenkins-agent/workspace/ci_linux'.  Did you enter a pattern relative to the correct directory?  Did you generate the XML report(s) for Cobertura?
00:45:49 [Cobertura] Skipped cobertura reports.
00:45:49 [xUnit] [INFO] - Starting to record.
00:45:49 [xUnit] [INFO] - Processing CTest-Version N/A (default)
00:45:49 [xUnit] [INFO] - [CTest-Version N/A (default)] - No test report file(s) were found with the pattern 'ws/build/*/Testing/*/Test.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'CTest-Version N/A (default)'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'CTest-Version N/A (default)'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'CTest' with the resolved pattern 'ws/build/*/Testing/*/Test.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing GoogleTest-1.6
00:45:49 [xUnit] [INFO] - [GoogleTest-1.6] - No test report file(s) were found with the pattern 'ws/build/*/test_results/**/*.gtest.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'GoogleTest-1.6'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'GoogleTest-1.6'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'GoogleTest' with the resolved pattern 'ws/build/*/test_results/**/*.gtest.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing JUnit
00:45:49 [xUnit] [INFO] - [JUnit] - No test report file(s) were found with the pattern 'ws/build/*/pytest.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'JUnit'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'JUnit'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'JUnit' with the resolved pattern 'ws/build/*/pytest.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing JUnit
00:45:49 [xUnit] [INFO] - [JUnit] - No test report file(s) were found with the pattern 'ws/build/*/test_results/**/*.xunit.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'JUnit'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'JUnit'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'JUnit' with the resolved pattern 'ws/build/*/test_results/**/*.xunit.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing CTest-Version N/A (default)
00:45:49 [xUnit] [INFO] - [CTest-Version N/A (default)] - No test report file(s) were found with the pattern 'work space/build space/*/Testing/*/Test.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'CTest-Version N/A (default)'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'CTest-Version N/A (default)'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'CTest' with the resolved pattern 'work space/build space/*/Testing/*/Test.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing GoogleTest-1.6
00:45:49 [xUnit] [INFO] - [GoogleTest-1.6] - No test report file(s) were found with the pattern 'work space/build space/*/test_results/**/*.gtest.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'GoogleTest-1.6'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'GoogleTest-1.6'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'GoogleTest' with the resolved pattern 'work space/build space/*/test_results/**/*.gtest.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing JUnit
00:45:49 [xUnit] [INFO] - [JUnit] - No test report file(s) were found with the pattern 'work space/build space/*/pytest.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'JUnit'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'JUnit'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'JUnit' with the resolved pattern 'work space/build space/*/pytest.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - Processing JUnit
00:45:49 [xUnit] [INFO] - [JUnit] - No test report file(s) were found with the pattern 'work space/build space/*/test_results/**/*.xunit.xml' relative to '/home/jenkins-agent/workspace/ci_linux' for the testing framework 'JUnit'.  Did you enter a pattern relative to the correct directory?  Did you generate the result report(s) for 'JUnit'?
00:45:49 [xUnit] [WARNING] - No test reports found for the metric 'JUnit' with the resolved pattern 'work space/build space/*/test_results/**/*.xunit.xml'.
00:45:49 [xUnit] [INFO] - Skipping the metric tool processing.
00:45:49 [xUnit] [INFO] - There are errors when processing test results.
00:45:49 [xUnit] [INFO] - Skipping tests recording.
00:45:49 Finished: SUCCESS

CI is stuck on virtualenv 16.7.9 until install-scripts option is honored.

ROS 2 Python packages which provide executables intended to be run via ros2 run $PACKAGENAME $EXENAME use the [install] install-scripts option of setup.cfg to install binaries into $base/lib/$PACKAGENAME, e.g. $base/lib/demo_nodes_py for the demo_nodes_py package. This intended behavior is thwarted when using virtualenv>20 or the venv module to create the virtual environment.

setuptools (and distutils) have code in place to detect when they're operating in a virtualenv and ignore installation-related options from setup.cfg entirely 1. The options are ignored whenever sys.prefix != sys.base_prefix. When using virtualenv<20, sys.prefix and sys.base_prefix both point to the virtualenv prefix. Using the venv module or virtualenv>=20, sys.base_prefix is the system python prefix (/usr in all tested cases). This is the case regardless of whether the --system-site-packages option is used or not.

This is a continuation of the work done in ros2/build_farmer#266, #385, and #399.

Some possible solutions

  1. We could stop using virtual environments in our CI entirely but this issue will also come up if any ROS users try to build packages in a workspace using virtual environments as well so without documentation it's not a total solution.

  2. We could make a case upstream that there should either be a way to opt-in to installation directory options in a virtual environment and provide patches to that end.

  3. We could enhance our build tools (i.e. colcon) to work around this by reading and re-providing these options via CLI where they are honored regardless.

  4. We could find some way to trick or force sys.base_prefix to match sys.prefix in new virtual environments, though I don't know what other behavior would be affected by such a change going forward.

Have CI report if a circular dependency is introduced

When we go to release a new distribution (like we are doing right now for Eloquent), we often find a couple of new circular dependencies have been introduced between repositories. Part of the reason for this is that there is no tool that tells us when this happens during the normal course of development.

What I'm thinking about is a tool that can tell us this during CI runs. The basic properties I'm thinking of are:

  1. Probably a colcon plugin (since it has the topological information)
  2. Run it at the end of CI, right after colcon test-result
  3. Have some sort of whitelist of "allowed" circular dependencies, since we solve some of them via release-time ignore files. This particularly happens with some of the "test" packages that we don't release into the distribution.

Having this tool would mean that we can find out about the circular dependencies at the time of their introduction, not several months later when we go to release.

Archives referencing skipped packages' local_setup.bat

Some packages included in ros2.repos are not built for windows, but they still seem to end up in the prefix.bat included in the fat archive. This causes the following output for users which it would be better to avoid if possible.

C:\Users\osrf\Downloads\ros2_package_ospl_home3\ros2-windows>call prefix.bat
not found: "C:\Users\osrf\Downloads\ros2_package_ospl_home3\ros2-windows\share/rttest/local_setup.bat"
not found: "C:\Users\osrf\Downloads\ros2_package_ospl_home3\ros2-windows\share/tlsf/local_setup.bat"
not found: "C:\Users\osrf\Downloads\ros2_package_ospl_home3\ros2-windows\share/tlsf_cpp/local_setup.bat"
not found: "C:\Users\osrf\Downloads\ros2_package_ospl_home3\ros2-windows\share/pendulum_control/local_setup.bat"

Linux CI doesn't install Connext when selecting rmw_connext_cpp

Bug:

When running a Linux CI job with --packages-up-to rmw_connext_cpp or --packages-select rmw_connext_cpp in the CI_ARGS variable, the script will erroneous choose not to install Connext for the CI job.

Expected behvaior:

When selecting rmw_connext packages in the CI job, Connext is correctly installed.

Further details:

The issue stems from the case state ment in the entry_point.sh file:

case "${CI_ARGS}" in

The glob is too loose in it's interpretation. It was originally built this way to accomodate the --ignore-rmw flag (e.g. export CI_ARGS=--do-venv --force-ansi-color --workspace-path /home/jenkins-agent/workspace/nightly_linux_release --ignore-rmw rmw_fastrtps_dynamic_cpp rmw_opensplice_cpp), but since CI_ARGS is one large environment variable, it matches other instances of rmw_connext_cpp.

Extend CI_BRANCH_TO_TEST to support ros2/ros2 repo

If we want to test a PR against ros2/ros2 (e.g. changing the ros2.repos file), we have to pass the custom repos file with the CI_ROS2_REPOS_URL parameter.

In my opinion in this context it would be good if the CI_BRANCH_TO_TEST also applied to the ros2/ros2 repo, not just the repos inside the ros2.repos file.

turtlebot jobs should run tests only on the packages from the turtlebot repos file

The turtlebot jobs, (nightly and CI) run tests for all packages in ros2.repos.

We could reduce the job times significantly if we were only running the tests of the packages on the turtlebot repos file.
This could also allow us to exercise the overlay logic if we were to split the logic into:

  • build a workspace with the ros2.repos file
  • build an overlay workspace with the turtlebot2.repos file
  • run tests on the overlay workspace
    (this proposal assumes that we still build everything from source)

OpenSplice not being found on Dosa.

I've pulled https://github.com/ADLINK-IST/opensplice/releases/download/OSPL_V6_7_171127OSS_RELEASE/PXXX-VortexOpenSplice-6.7.171127OSS-HDE-x86_64.darwin10_clang-release-installer.tar.gz onto to Dosa and exported OSPL_HOME to in the .bashrc.

This is sufficient for builds in a local ROS2 workspace to pick up and successfully build rmw_opensplice_cpp but when trying to run CI with opensplice I get the following failure http://ci.ros2.org/job/test_ci_osx/31/

It could be that CI either doesn't pick up variables set in .bash_profile/.bashrc or that its filtering them out.

There are local test failures when using this latest opensplice version so I don't think sorting this out is a priority for the current release window.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.