Coder Social home page Coder Social logo

centos-paas-sig / ci-pipeline Goto Github PK

View Code? Open in Web Editor NEW
23.0 13.0 35.0 5.06 MB

There are many projects that are concentrated on CI for Atomic Openshift. This is focused on complete CI testing of all Atomic and Base Runtime components for a new Atomic deliverable that comes from Fedora sources.

License: GNU General Public License v3.0

Python 14.88% Shell 38.65% Makefile 0.14% Groovy 37.23% Dockerfile 9.10%

ci-pipeline's Introduction

Table of Contents

CI-Pipeline Architecture and Design

CI-Pipeline

What Does CI/CD Mean in the Context of the CI-Pipeline Project?

True CI starts at gating developers before code is integrated into the general code base to find bugs sooner. It is less costly for the developer that changed the code to fix their own issues before the code is merged into the repo. Obviously there is a certain balance of what testing/validation is run and time to get feedback to the developer and this may vary among projects.

CI-Pipeline Overview

CI in general is the continuous integration of components/projects/products and validating these work together and then providing feedback to developers. Continuous delivery is the act of taking these integrated components that have been validated and are stable and delivering them with consistency over and over efficiently. Continuous deployment allows us to take the deliverable and deploy it on some set of infrastructure to be used in a development, stage, and/or production environments.

The way continuous integration works is that robots test every change. They can then tell you which change caused the integrated result to break. We can then “gate” that change and ask a human to fix it. When those changes are small and regular, the robots can gate it, and it’s trivial for a developer to quickly see why and how the change broke the code. On the other hand, if the changes that flow into continuous integration are massive bundles of changes, with months between them, with myriad test failures, then it becomes a tedious never-ending process to dissect them and figure out what went wrong. The effect of continuous integration evaporates. Hence we arrive at the following truth:

The closer to the source of change the continuous integration acts, the more effective it is.

This CI/CD toolset is designed to provide a flexible solution to that problem. The pipeline is fully containerized and built using upstream projects Openshift, Jenkins, and Ansible. This makes the pipeline extremely portable and can run in any Openshift instance and adapted to any project/product.

Dependencies and Assumptions

Automation Frameworks and Tests

The CI-Pipeline is NOT an automation framework or set of automated tests. There are plenty of those that exist and this project's goal is to be flexible to accommodate execution of any automation framework, tests and tools, especially if they are already integrated in a container.
Unit tests can easily be defined as make targets and can run as part of the build and need minimal resources. Functional and integration testing may require more robust test beds to execute scenarios.
In most cases these more complex tests can still be verified inside containers or VMs running inside containers. This is both more efficient and provides a quicker feedback loop for developers.

Triggering the Pipeline

Since Jenkins is integrated with many trigger options we can easily trigger the pipeline based on cron, github, gerrit, gitlab, and fedmsg thanks to the Jenkins JMS plugin

CI/CD Toolset and Infrastructure

We are dependent on the following tools:

  • Openshift
    • Containers
  • Jenkins 2.0
    • Plugins
    • Pipeline
    • Shared libraries
  • Ansible

Openshift, Jenkins 2.0 pipeline, Pipeline Shared Libraries

Openshift

The CI-Pipeline is defined using Jenkins 2.0 pipeline (Jenkinsfile) + shared libraries. Jenkins 2.0 Pipeline is already integrated into Openshift (Openshift Pipelines Deep Dive). All our pipeline components, containers, etc. are running inside the CentOS Openshift instance.

Jenkins 2.0 Pipeline

Our Jenkins master that runs inside Openshift is Jenkins Master

Jenkins Masters and Slaves

The Jenkins Master is created from Openshift's s2i (source-to-image) feature. This means we can create our infrastructure for our pipeline right from source. The Jenkins Master is backed by a persistent volume to maintain historical data.

Slave images are created from Dockerfiles.

├── config
│   └── s2i
│       ├── create-containers.sh
│       ├── jenkins
│       │   ├── continuous-infra-create-openshift-project
│       │   ├── continuous-infra-jenkins-master-s2i-template.yaml
│       │   ├── continuous-infra-slave-template.yaml
│       │   ├── master
│       │   │   ├── configuration
│       │   │   │   ├── hudson.plugins.ircbot.IrcPublisher.xml
│       │   │   │   ├── hudson.tasks.Maven.xml
│       │   │   │   ├── init.groovy
│       │   │   │   └── jobs
│       │   │   │       ├── fedmsg-test
│       │   │   │       │   └── config.xml
│       │   │   │       └── maven-test
│       │   │   │           └── config.xml
│       │   │   └── plugins.txt
│       │   └── slave
│       │       └── Dockerfile
Pipeline Shared Libraries

We have shared libraries that are used in this repo/project and we also use other shared libraries that we contribute to along with the CentOS Community
cico-pipeline-library

The libraries in this repo/project follows the Jenkins community shared library structure:

├── src
│   └── org
│       └── centos
│           └── pipeline
│               └── PipelineUtils.groovy
└── vars
    └── pipelineUtils.groovy

CI the CI-Pipeline - Inception

Since we can easily integrate into github then we can use our own pipeline to validate our stage containers, pipeline code, and shared libraries.

  1. We inspect the changelog in the pipeline using the Jenkins declaritive pipeline DSL.
  2. If stage containers are touched then they are built in Openshift and validated
  3. If the stage container passes validation it is tagged stable and promoted to production in the pipeline
  4. If a container and/or pipeline code and/or shared libraries are changed the change is run through the pipeline for validation

Example of CI the CI-Pipeline

ci-the-ci-pipeline

Pipeline References

Currently documentation, Jenkins master/slave containers, container definitions - s2i/docker, pipeline shared libraries, tools, etc. are located here in this repo.

Example Project - Fedora Atomic Host

In leveraging the CI-Pipeline this project was able to deliver a stable Fedora Atomic host and gate packagers in Fedora. We were able to integrate with fedmsg as our trigger and publishing communication backbone. This allowed us to gate packagers in the Fedora infrastructure.

Fedora Atomic Host Complete View

Automation Frameworks and Testing in Fedora

Setup and invoking of tests is being done using ansible as outlined in Invoking CI Tests

Design Diagrams

ci-pipeline-detail

CI-Pipeline in Action

Fedora Atomic f26 example

Fedora Atomic Host Pipeline Stages

Trigger

Once packages are pushed to Fedora dist-git this will trigger a message. The pipeline will be triggered via the Jenkins JMS plugin for dist-git messages on fedmsg.
Only Fedora Atomic Host packages are monitored for changes currently. This can easliy be broadened to all of Fedora packages.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.package.[queued,ignored].

Build Package

Once the pipeline is triggered as part of the build process if unit tests exist they will be executed.

The end result is package will be produced to then be used for further testing. Success or failure will result with a fedmsg back to the Fedora package maintainer.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.package.[running,complete].

Functional Tests on Packages

Functional tests will be executed on the produced package from the previous stage of the pipeline if they exist. This will help identify issues isolated to the package themselves. Success or failure will result with a fedmsg back to the Fedora package maintainer.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.package.test.functional.[queued,running,complete].

Compose an OStree

If functional tests are successful in the previous stage of the pipeline then an OStree compose is generated.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.compose.[running,complete].

Integration Tests on OStree Composes and Images

Integration tests are run on the OStree compose. Success or failure will result with a fedmsg back to the Fedora package maintainer.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.compose.test.integration[queued,running,complete].

Image Generated From Successful Integration Tests On OStree

An image will be initially generated at a certain interval when there has been successful integration test execution on an OStree compose. Success or failure will result with a fedmsg back to the Fedora package maintainer.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.image.[running,complete].

Openshift Cluster and e2e Conformance tests on top of Fedora Atomic Host

If integration tests of the images are successful an openshift cluster will be configured using the Atomic Openshift Installer with the new Fedora Atomic Host image as the base system. Once the cluster is configured Kubernetes conformance tests will run. Success or failure will result with a fedmsg back to the Fedora package maintainer.

Image Smoke Test Validation

The validation left is to make sure the image can boot and more smoke tests may follow if required. Success or failure will result with a fedmsg sent back to the Fedora package maintainer. Also, this can trigger the Red Hat continuous delivery process to run more comprehensive testing.

CI Pipeline messages sent via fedmsg for this stage are captured by the topics org.centos.prod.ci.pipeline.image.test.smoke.[running,complete].

fedmsg Bus

Communication between Fedora, CentOS, and Red Hat infrastructures will be done via fedmsg. Messages will be received of updates to dist-git repos that we are concerned about for Fedora Atomic host. Triggering will happen from Fedora dist-git. The CI-Pipeline in CentOS infrastructure will build and functional test packages, compose and integration test ostrees, generate and smoke test (boot) images. We are dependant on CentOS Infrastructure for allowing us a hub for publishing messages to fedmsg.

fedmsg - Message Types

Below are the different message types that we listen and publish. There will be different subtopics so we can keep things organized under the org.centos.prod.ci.pipeline.* umbrella. The fact that ‘org.centos’ is contained in the messages is a side effect of the way fedmsg enforces message naming.

fedmsg - Message Legend
  • CI_TYPE - Type of message we are sending (default value = custom)
    • ex. custom
  • build_id - Jenkins pipeline build ID
    • ex. 91
  • build_url - Full url to the blue ocean Jenkins pipeline build view
  • username - Audit of who is sending the message (default value = fedora-atomic)
    • ex. fedora_atomic
  • compose_url - Full url to the OStree compose
  • compose_rev - SHA-1 of the compose
    • ex. c15b07cc0f560dd4325b64b8ed8f5750ab102f2ff2e0ace39cda4158f900e4da
  • rev - This is the commit SHA-1 that is passed on from the dist-git message we recieve
    • ex. 591b0d2fc67a45e4ad13bdc3e312d5554852426a
  • image_url - Full url to the image
  • image_name - Image name
    • ex. fedora-atomic-26.200-c15b07cc0f560dd.qcow2
  • message-content - If additional content is needed in the bod of the message (default = "")
  • namespace - Packaging type passed from dist-git message
    • ex. rpms
  • original_spec_nvr - The NVR of the RPM if the spec file hadn't been modified
    • ex. gnutls-3.5.15-1.fc26
  • nvr - The NVR of the RPM after modification, so including sha and # commits
    • ex. gnutls-3.5.15-1.246.a3e666c.fc26
  • CI_NAME - Jenkins pipeline job name
    • ex. ci-pipeline-f26
  • repo - Package name
    • ex. vim
  • topic - Topic that is being published on the fedmsg bus
    • ex. org.centos.prod.ci.pipeline.image.complete
  • status - Status of the stage and overall pipeline at the time when the message is published. UNSTABLE status indicates that some tests did not pass and should be analyzed.
    • ex. SUCCESS
    • options = <SUCCESS/FAILURE/UNSTABLE/ABORTED>
  • branch - Fedora branch master = rawhide for now this may change in the future
    • ex. f26
  • type - Image type
    • ex. qcow2
  • test_guidance - required by downstream CI
    • ex. "''"
  • ref - Indication of what we are building distro/branch/arch/distro_type
    • ex. fedora/f26/x86_64/atomic-host
  • msgJson - A full dump of the message properties in one field of JSON

Each change passing through the pipeline is uniquely identified by repo, rev, and namespace.

Trigger - org.fedoraproject.prod.git.receive

Below is an example of the message that we will trigger off of to start our CI pipeline. We concentrate on the commit part of the message.

username=jchaloup
stats={u'files': {u'fix-rootScopeNaming-generate-selfLink-issue-37686.patch': {u'deletions': 8, u'additions': 8, u'lines': 16}, u'build-with-debug-info.patch': {u'deletions': 8, u'additions': 8, u'lines': 16}, u'get-rid-of-the-git-commands-in-mungedocs.patch': {u'deletions': 25, u'additions': 0, u'lines': 25}, u'kubernetes.spec': {u'deletions': 13, u'additions': 11, u'lines': 24}, u'remove-apiserver-add-kube-prefix-for-hyperkube-remov.patch': {u'deletions': 0, u'additions': 169, u'lines': 169}, u'.gitignore': {u'deletions': 1, u'additions': 1, u'lines': 2}, u'sources': {u'deletions': 1, u'additions': 1, u'lines': 2}, u'remove-apiserver-add-kube-prefix-for-hyperkube.patch': {u'deletions': 66, u'additions': 0, u'lines': 66}, u'use_go_build-is-not-fully-propagated-so-make-it-fixe.patch': {u'deletions': 5, u'additions': 5, u'lines': 10}, u'Hyperkube-remove-federation-cmds.patch': {u'deletions': 118, u'additions': 0, u'lines': 118}, u'fix-support-for-ppc64le.patch': {u'deletions': 9, u'additions': 9, u'lines': 18}}, u'total': {u'deletions': 254, u'files': 11, u'additions': 212, u'lines': 466}}
name=Jan Chaloupka
namespace=rpms
rev=b0ef5e0207cea46836a49cd4049908f14015ed8d
agent=jchaloup
summary=Update to upstream v1.6.1
repo=kubernetes
branch=f26
path=/srv/git/repositories/rpms/kubernetes.git
seen=False
message=Update to upstream v1.6.1- related: #1422889
[email protected]

Dist-git message example

{
  "source_name": "datanommer",  
  "i": 1, 
  "timestamp": 1493386183.0, 
  "msg_id": "2017-b29fa2b4-0600-4f08-9475-5f82f6684bd4", 
  "topic": "org.fedoraproject.prod.git.receive", 
  "source_version": "0.6.5", 
  "signature": "MbQSb1uwzh6UIFKVm+Uxt+56nW/QRH1nOehifxUrbZfiEDEscRdHtb8dj1Skdv7fcZGHhNlR3PGI\nz/4YqPFJjoAM/k60FsnBIIG1gklJaFBM8MloEYauzo/fUK//W99ojk3UPK0lGTIBijG2knbD9t3T\nUMRuDjt45zmGBXHPlR8=\n", 
  "msg": {
    "commit": {
      "username": "trasher", 
      "stats": {
        "files": {
          "sources": {
            "deletions": 1, 
            "additions": 1, 
            "lines": 2
          }, 
          "php-simplepie.spec": {
            "deletions": 5, 
            "additions": 8, 
            "lines": 13
          }, 
          ".gitignore": {
            "deletions": 0, 
            "additions": 1, 
            "lines": 1
          }
        }, 
        "total": {
          "deletions": 6, 
          "files": 3, 
          "additions": 10, 
          "lines": 16
        }
      }, 
      "name": "Johan Cwiklinski", 
      "rev": "81e09b9c83e8550b54a64c7bdb4e5d7b534df058", 
      "namespace": "rpms", 
      "agent": "trasher", 
      "summary": "Last upstream release", 
      "repo": "php-simplepie", 
      "branch": "f24", 
      "seen": false, 
      "path": "/srv/git/repositories/rpms/php-simplepie.git", 
      "message": "Last upstream release\n", 
      "email": "[email protected]"
    }
  }
}

org.centos.prod.ci.pipeline.package.ignore

{
  "i": 1, 
  "timestamp": 1507135702, 
  "msg_id": "2017-1dc328f8-7421-4d09-8a4f-41ec82f00785", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.package.ignore", 
  "msgJson": "{\"CI_NAME\":\"ci-pipeline-trigger\",\"CI_TYPE\":\"custom\",\"branch\":\"rawhide\",\"build_id\":\"52867\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos.org/jo
b/ci-pipeline-trigger/52867/\",\"message-content\":\"\",\"namespace\":\"rpms\",\"ref\":\"fedora/rawhide/x86_64/atomic-host\",\"repo\":\"polari\",\"rev\":\"8073b99c61e58abb8fbcd60cf8ea518d184d0108\",\"status\":\"SUCCESS\",\"test_guidance\"
:\"''\",\"topic\":\"org.centos.prod.ci.pipeline.package.ignore\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "52867", 
    "username": "fedora-atomic", 
    "rev": "8073b99c61e58abb8fbcd60cf8ea518d184d0108", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-trigger/52867/", 
    "namespace": "rpms", 
    "CI_NAME": "ci-pipeline-trigger", 
    "repo": "polari", 
    "topic": "org.centos.prod.ci.pipeline.package.ignore", 
    "status": "SUCCESS", 
    "branch": "rawhide", 
    "test_guidance": "''", 
    "ref": "fedora/rawhide/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.package.queued

{
  "i": 1, 
  "timestamp": 1507062395, 
  "msg_id": "2017-2d2e013c-12bd-4ee7-9edc-167aaf0a162b", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.package.queued", 
  "msgJson": "{\"CI_NAME\":\"ci-pipeline-trigger\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"52257\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos.org/job/ci
-pipeline-trigger/52257/\",\"message-content\":\"\",\"namespace\":\"rpms\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"163e745ef5f654a5df09ccca9350dc98e926af39\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"
topic\":\"org.centos.prod.ci.pipeline.package.queued\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "52257", 
    "username": "fedora-atomic", 
    "rev": "163e745ef5f654a5df09ccca9350dc98e926af39", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-trigger/52257/", 
    "namespace": "rpms", 
    "CI_NAME": "ci-pipeline-trigger", 
    "repo": "mesa", 
    "topic": "org.centos.prod.ci.pipeline.package.queued", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.package.running

{
  "i": 1, 
  "timestamp": 1507135699, 
  "msg_id": "2017-d7510906-71dd-48a9-a3b4-56d6f64ce9c9", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.package.running", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"\",\"original_spec_nvr\":\"\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0
292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.package.running\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "", 
    "username": "fedora-atomic", 
    "nvr": "", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "", 
    "topic": "org.centos.prod.ci.pipeline.package.running", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.package.complete

{
  "i": 1, 
  "timestamp": 1507136310, 
  "msg_id": "2017-0f165bf3-0324-40cc-9cd4-a008c9522d96", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.package.complete", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86
_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.package.complete\",\"username\":\"fedora-atomic\"}"
, 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "", 
    "topic": "org.centos.prod.ci.pipeline.package.complete", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.package.test.functional.queued

{
  "i": 1,
  "timestamp": 1501741048,
  "msg_id": "2017-b420134c-0e39-4f70-8e5f-0975d7019e4b",
  "crypto": "x509",
  "topic": "org.centos.prod.ci.pipeline.package.test.functional.queued",
  "msg": {
    "CI_TYPE": "custom",
    "build_id": "91",
    "username": "fedora-atomic",
    "rev": "591b0d2fc67a45e4ad13bdc3e312d5554852426a",
    "message-content": "",
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/91/pipeline/",
    "namespace": "rpms",
    "CI_NAME": "ci-pipeline-f26",
    "repo": "vim",
    "original_spec_nvr": "vim-7.4.160-1.fc26",
    "nvr": "vim-7.4.160-1.200.591b0d2.fc26",
    "topic": "org.centos.prod.ci.pipeline.package.test.functional.queued",
    "status": "SUCCESS",
    "test_guidance": "''",
    "branch": "f26",
    "package_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/repo/vim_repo/",
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.package.test.functional.running

{
  "i": 1,
  "timestamp": 1501741048,
  "msg_id": "2017-b420134c-0e39-4f70-8e5f-0975d7019e4b",
  "crypto": "x509",
  "topic": "org.centos.prod.ci.pipeline.package.test.functional.running",
  "msg": {
    "CI_TYPE": "custom",
    "build_id": "91",
    "username": "fedora-atomic",
    "rev": "591b0d2fc67a45e4ad13bdc3e312d5554852426a",
    "message-content": "",
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/91/pipeline/",
    "namespace": "rpms",
    "CI_NAME": "ci-pipeline-f26",
    "repo": "vim",
    "original_spec_nvr": "vim-7.4.160-1.fc26",
    "nvr": "vim-7.4.160-1.200.591b0d2.fc26",
    "topic": "org.centos.prod.ci.pipeline.package.test.functional.running",
    "status": "SUCCESS",
    "test_guidance": "''",
    "branch": "f26",
    "package_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/repo/vim_repo/",
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.package.test.functional.complete

{
  "i": 1,
  "timestamp": 1501741048,
  "msg_id": "2017-b420134c-0e39-4f70-8e5f-0975d7019e4b",
  "crypto": "x509",
  "topic": "org.centos.prod.ci.pipeline.package.test.functional.complete",
  "msg": {
    "CI_TYPE": "custom",
    "build_id": "91",
    "username": "fedora-atomic",
    "rev": "591b0d2fc67a45e4ad13bdc3e312d5554852426a",
    "message-content": "",
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/91/pipeline/",
    "namespace": "rpms",
    "CI_NAME": "ci-pipeline-f26",
    "repo": "vim",
    "original_spec_nvr": "vim-7.4.160-1.fc26",
    "nvr": "vim-7.4.160-1.200.591b0d2.fc26",
    "topic": "org.centos.prod.ci.pipeline.package.test.functional.complete",
    "status": "SUCCESS",
    "test_guidance": "''",
    "branch": "f26",
    "package_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/repo/vim_repo/",
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

org.centos.prod.ci.pipeline.compose.running

{
  "i": 1, 
  "timestamp": 1507136315, 
  "msg_id": "2017-809b57f5-581d-4270-8cdb-c0ced8d37cc9", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.compose.running", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"''\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2
.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.
prod.ci.pipeline.compose.running\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "''", 
    "topic": "org.centos.prod.ci.pipeline.compose.running", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.compose.complete

{
  "i": 1, 
  "timestamp": 1507136621, 
  "msg_id": "2017-6723f16f-93e1-4ec0-a38e-78f092a592ee", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.compose.complete", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"message-content\":\"\",\"n
amespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\
":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.compose.complete\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.compose.complete", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.image.running

{
  "i": 1, 
  "timestamp": 1507136627, 
  "msg_id": "2017-9f9bde8a-0508-44cc-9ae3-a962e9228125", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.image.running", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"image_name\":\"''\",\"imag
e_url\":\"''\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a
0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.image.running\",\"type\":\"qcow2\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "image_name": "''", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.image.running", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "type": "qcow2", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "image_url": "''", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.image.complete

{
  "i": 1, 
  "timestamp": 1507137242, 
  "msg_id": "2017-b6db02c4-2e0d-4535-bb3c-6c771cc1f20e", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.image.complete", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"image_name\":\"fedora-atom
ic-26.391-9a4e37c459e67c7.qcow2\",\"image_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/images/fedora-atomic-26.391-9a4e37c459e67c7.qcow2\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc2
6.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.cent
os.prod.ci.pipeline.image.complete\",\"type\":\"qcow2\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "image_name": "fedora-atomic-26.391-9a4e37c459e67c7.qcow2", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.image.complete", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "type": "qcow2", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "image_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/images/fedora-atomic-26.391-9a4e37c459e67c7.qcow2", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.image.test.smoke.running

{
  "i": 1, 
  "timestamp": 1507137248, 
  "msg_id": "2017-d27543a1-53bf-46f6-87c5-8b01258fc112", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.image.test.smoke.running", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"image_name\":\"fedora-atom
ic-26.391-9a4e37c459e67c7.qcow2\",\"image_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/images/fedora-atomic-26.391-9a4e37c459e67c7.qcow2\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc2
6.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.cent
os.prod.ci.pipeline.image.test.smoke.running\",\"type\":\"qcow2\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "image_name": "fedora-atomic-26.391-9a4e37c459e67c7.qcow2", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.image.test.smoke.running", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "type": "qcow2", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "image_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/images/fedora-atomic-26.391-9a4e37c459e67c7.qcow2", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.image.test.smoke.complete

{
  "i": 1, 
  "timestamp": 1507137602, 
  "msg_id": "2017-0772207c-3664-427e-8721-67bec95cd373", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.image.test.smoke.complete", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"image_name\":\"fedora-atom
ic-26.391-9a4e37c459e67c7.qcow2\",\"image_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/images/fedora-atomic-26.391-9a4e37c459e67c7.qcow2\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc2
6.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.cent
os.prod.ci.pipeline.image.test.smoke.complete\",\"type\":\"qcow2\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "image_name": "fedora-atomic-26.391-9a4e37c459e67c7.qcow2", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.image.test.smoke.complete", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "type": "qcow2", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "image_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/images/fedora-atomic-26.391-9a4e37c459e67c7.qcow2", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.compose.test.integration.queued

{
  "i": 1, 
  "timestamp": 1507138147, 
  "msg_id": "2017-4ef1db00-ef4b-4b3f-ac54-a79866f24d62", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.compose.test.integration.queued", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"message-content\":\"\",\"n
amespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\
":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.compose.test.integration.queued\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.compose.test.integration.queued", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.compose.test.integration.running

{
  "i": 1, 
  "timestamp": 1507138153, 
  "msg_id": "2017-9a9781b9-3d17-438c-b22d-002d6e505428", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.compose.test.integration.running", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"message-content\":\"\",\"n
amespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\
":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.compose.test.integration.running\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.compose.test.integration.running", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.compose.test.integration.complete

{
  "i": 1, 
  "timestamp": 1507138177, 
  "msg_id": "2017-d104ac21-0807-4fa8-b8db-8905d11aadfe", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.compose.test.integration.complete", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"compose_url\":\"http://artifacts.ci.centos.org/fedora-atomic/f26/ostree\",\"message-content\":\"\",\"n
amespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"original_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\
":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeline.compose.test.integration.complete\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "compose_url": "http://artifacts.ci.centos.org/fedora-atomic/f26/ostree", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.compose.test.integration.complete", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src"
  }
}

org.centos.prod.ci.pipeline.complete

{
  "i": 1, 
  "timestamp": 1507138233, 
  "msg_id": "2017-203ce174-de23-4e62-9ddc-3522b59d62a1", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.complete", 
  "msgJson": "{\"CI_NAME\":\"continuous-infra-ci-pipeline-f26\",\"CI_TYPE\":\"custom\",\"branch\":\"f26\",\"build_id\":\"464\",\"build_url\":\"https://jenkins-continuous-infra.apps.ci.centos
.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/\",\"compose_rev\":\"9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04\",\"message-content\":\"\",\"namespace\":\"rpms\",\"nvr\":\"mesa-17.1.5-1.816.35fdde9.fc26.2.src\",\"origi
nal_spec_nvr\":\"mesa-17.1.5-1.fc26.2\",\"ref\":\"fedora/f26/x86_64/atomic-host\",\"repo\":\"mesa\",\"rev\":\"35fdde92a0292def6a99acd3c237272789d020b2\",\"status\":\"SUCCESS\",\"test_guidance\":\"''\",\"topic\":\"org.centos.prod.ci.pipeli
ne.complete\",\"username\":\"fedora-atomic\"}", 
  "msg": {
    "CI_TYPE": "custom", 
    "build_id": "464", 
    "original_spec_nvr": "mesa-17.1.5-1.fc26.2", 
    "username": "fedora-atomic", 
    "nvr": "mesa-17.1.5-1.816.35fdde9.fc26.2.src", 
    "rev": "35fdde92a0292def6a99acd3c237272789d020b2", 
    "message-content": "", 
    "build_url": "https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/464/pipeline/", 
    "namespace": "rpms", 
    "CI_NAME": "continuous-infra-ci-pipeline-f26", 
    "repo": "mesa", 
    "compose_rev": "9a4e37c459e67c7187df4c571ddd0ad75969633d40bdfaa7c1d95bbff5fe1e04", 
    "topic": "org.centos.prod.ci.pipeline.complete", 
    "status": "SUCCESS", 
    "branch": "f26", 
    "test_guidance": "''", 
    "ref": "fedora/f26/x86_64/atomic-host"
  }
}

ci-pipeline's People

Contributors

14rcole avatar andrei-stepanov avatar arilivigni avatar bgoncalv avatar billrainford avatar bookwar avatar cevich avatar dirgim avatar herlo avatar ignatenkobrain avatar jimbair avatar jlebon avatar joejstuart avatar johnbieren avatar p3ck avatar psss avatar pypingou avatar robnester-rh avatar samvarankashyap avatar scoheb avatar sergturi avatar thrix avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

arilivigni p3ck samvarankashyap dperpeet scoheb sm00th billrainford jlebon johnbieren pypingou cevich stefwalter andrei-stepanov goern thrix psiroky rsk 14rcole herlo firemanxbr tflink vibe13 eduardocerqueira tstellar dirgim bgoncalv ignatenkobrain psss xlogin mike-nguyen akondasif bookwar kshithijiyer jimbair wanghekai

ci-pipeline's Issues

Multiple package.running

I ran into these three messages:

Is this expected?

Tests must be invoked with a qcow2 test subject

The tests must be invoked with a test subject, and not a running Atomic Host.

Each change (package or otherwise) composed in the pipeline should prepare a test subject (ie: qcow):

https://fedoraproject.org/wiki/Changes/InvokingTests

The tests are invoked with that test subject, which will then launch and/or consume it as appropriate. Multiple techniques are available to do this, among which might be:

  • Recomposing a qcow2 each time (expensive)
  • Running an rpm-ostree rebase inside an existing qcow2
  • Running an rpm-ostree install inside an existing qcow2

virt-customize installs the kernel, but it's not running the post-transaction scripts

The kernel CI test fails early on when it checks the kernel version. I then downloaded the qcow2 image, booted it up, and discovered it's not booting the newly installed kernel. It looks like the kernel's post transaction script (which runs kernel-install add <kernel version>, generating the initramfs and grub entry) isn't being run.

I used virt-customize locally on the F28 cloud image to install the same kernel and it was properly installed, so it's not immediately obvious to me what's causing it to not happen in the CI environment.

What does unstable mean?

Looking at this pipeline run, it seems to have passed but has been marked as unstable: https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/continuous-infra-ci-pipeline-f26/detail/continuous-infra-ci-pipeline-f26/231/pipeline

what does this mean?

I found this in the logs:

[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing JUnit
[xUnit] [INFO] - [JUnit] - 2 test report file(s) were found with the pattern '**/logs/ansible_xunit.xml' relative to '/tmp/workspace/continuous-infra-ci-pipeline-f26' for the testing framework 'JUnit'.
[xUnit] [INFO] - Check 'Failed Tests' threshold.
[xUnit] [INFO] - The total number of tests for this category exceeds the specified 'unstable' threshold value.
[xUnit] [INFO] - Setting the build status to UNSTABLE
[xUnit] [INFO] - Stopping recording.

but it's not quite helping me.

Any idea?
Thanks :)

Message documentation vs production

For the fedmsg client of ccsdb (http://resultsdb.ci.centos.org/ccsdb/) I based myself on the documentated format for the fedmsg messages present in the readme file [1] of this project.
I am now seeing some messages being published that differ quite a bit from the announce format, so I am curious if the current format is temporary or final. If the later then we should adjust our documentation.

Expected:

{ 
  u'i': 1,
  u'msg': { u'build_url': u'<full-jenkins-url-to-build>',
            u'package_url': u'<full-url-to-package>',
            u'build_id': <Jenkins-build-id-number>,
            u'ref': u'fedora/rawhide/${basearch}/atomic-host',
      	u'rev': u'<sha of the commit from dist-git>', 
      	u'namespace': u'rpms', 
      	u'repo': u'php-simplepie',
            u'status': u'<success>',
            u'test_guidance': u'<comma-separated-list-of-test-suites-to-run>'},
  u'msg_id': u'2015-4b5aae66-b713-4c22-bb4a-1277d4402375',
  u'timestamp': 1379638157.759283,
  u'topic': u'org.centos.prod.ci.pipeline.package.complete',
  u'username': u'fedora-atomic'
}

Observed:

{
  "i": 1, 
  "timestamp": 1496243297, 
  "msg_id": "2017-71d0a514-7925-4e62-9a92-19c1ebdc352d", 
  "crypto": "x509", 
  "topic": "org.centos.prod.ci.pipeline.package", 
  "msg": {
    "CI_TYPE": "custom", 
    "username": "fedora-atomic", 
    "message-content": "{\n  \"build_url\": \"https://ci.centos.org/job/ci-pipeline-rpmbuild-trigger/17354/\",\n  \"build_id\": \"17354\",\n  \"ref\": \"fedora/f25/x86_64/atomic-host\",\n  \"rev\": \"578ea71331a63e67b8681c02638af400d76d6611\",\n  \"namespace\": \"rpms\",\n  \"repo\": \"krita\",\n  \"status\": \"success\",\n  \"test_guidance\": \"\"\n}", 
    "CI_NAME": "ci-pipeline-rpmbuild-trigger", 
    "topic": "org.centos.prod.ci.pipeline.package.ignore", 
    "CI_STATUS": "passed"
  }
}

See that the topic is not consistent in the outer envelop versus the inner one, the CI_TYPE, CI_NAME, CI_STATUS and messages-content fields are unexpected.

I am fine with adjusting my code but I just would like to know if I should or if the format going to be changed to fit the documentation.

Thanks!

[1] https://github.com/CentOS-PaaS-SIG/ci-pipeline/#message-types

Include the NVR in the pipeline messages

In order to integrate the CI results in a way that greenwave and bodhi can use, we would need to store in resultsdb the nvr extracted straight from the spec file.

Potentially this may be as simple as:

$ rpmspec -q guake.spec  
guake-0.8.8-3.fc25.noarch

(rpmspec is part of rpm-build)

We'll then store this value in resultsdb and when bodhi will ask for the CI results concerning this build greenwave will be able to retrieve them.

Thanks

Include branch information?

Looking at http://resultsdb.ci.centos.org/ccsdb/api/status/7ac01ab175e7ddb7f3e2a9e8d0d1f26a2b953673 I see that the entry was created on date_created "2017-06-01T14:13:32.414641" but looking into the git log, I see:

commit 7ac01ab175e7ddb7f3e2a9e8d0d1f26a2b953673
...
Date:   Fri Feb 10 05:24:49 2017 +0000

This surprises me a little bit, the commit is months old but got into the CI pipeline recently, so I believe it got pushed into a release branch (f26, f25 or f24 since they all seem to be at the same commit now). In order to clear these situation it might be interested to keep the branch of the commit through the pipeline.

Is this doable? Hard to do?

Fedora Atomic CI - Fedora 27

I'm not sure I'm reading the config files correctly, but looking at the logs (e.g. for vim), it looks like the pipeline is running tests on Fedora Atomic Host 26.

Can we switch this to 27?

virt-customize does funny things with yum

  • It passes dnf options to yum
  • It adds second repo god knows why
Added rust-xattr repo from file:///etc/yum.repos.d/rust-xattr

Fedora - Rawhide - Developmental packages for t  12 MB/s |  62 MB     00:05    

test-rust-xattr                                 1.3 MB/s | 1.5 kB     00:00    

rust-xattr                                      1.4 MB/s | 1.5 kB     00:00    

Last metadata expiration check: 0:00:00 ago on Fri Jul 20 10:19:24 2018.

Dependencies resolved.

================================================================================

 Package              ArchVersion                                          Repository      Size

================================================================================

Installing:

 rust-xattr-devel     noarch0.2.2-3.fc29.pr.a9347763bfb04006beb0b694bc9195bd test-rust-xattr 22 k

FedMsgs to signal downstream CI missing fields

Hi Guys,

Giulio Frasca on the downstream CI team reports that we received several CI FedMsgs today that are missing critical fields.

Below is a link to and the output of one of our legacy production jenkins jobs -- a simple job that listens for the message that would trigger a downstream CI run and emails our team the contents of the message. (We received several messages today that matched the one below). That job is listening to tcp://hub.fedoraproject.org:9940

We received 11 messages today from upstream with this issue. Before today's flurry of messages the last message and last valid org.centos.prod.ci.pipeline.image.test.smoke.complete message was on Sept 19th.

The messages seem to be missing 3 of the critical fields we use to validate one of this messages -- 'image_name', 'image_url', 'type'

Here is the full list of FedMsg fields we validate in the downstream CI pipeline: (Basically the minimum set I need to care about, more/other fields don't hurt, but if I don't get the list below we don't proceed with the pipeline)

'CI_NAME',
'CI_TYPE',
'status',
'topic',
'username',
'image_name',
'image_url',
'type']

With the above issue it blocks downstream CI's ability to run in reaction to what upstream is producing (As I don't know where to find the bits to test from this malformed messages.)

If I can provide any additional info, please let me know.

Thanks,
-Bill

PASTED INFO AND JOB OUTPUT:
This email was generated by the following Jenkins job build: https://rhcs-stage-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DASHost-message-bus-listener-8782d/69/

The message has following header info:
CI TYPE: custom
Status: SUCCESS
Topic: org.centos.prod.ci.pipeline.image.test.smoke.complete
User Name: fedora-atomic

The body of the message is:
{"CI_TYPE":"custom",
"build_id":"419",
"original_spec_nvr":"dnsmasq-2.76-4.f26",
"username":"fedora-atomic",
"nvr":"dnsmasq-2.76-4.126.1bca83a.fc26.x86_64",
"rev":"1bca83a5d36c969c4e8c2c2343d592e57094903d",
"message-content":"",
"build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/job/continuous-infra-ci-pipeline-f26/419/",
"namespace":"rpms",
"CI_NAME":"continuous-infra-ci-pipeline-f26",
"repo":"dnsmasq",
"compose_rev":"a1c75f154ebf755810b3463141324df9a6f9ab47aff4c10a6f57b3d6df445bdd",
"topic":"org.centos.prod.ci.pipeline.image.test.smoke.complete",
"status":"SUCCESS",
"branch":"f26",
"test_guidance":"''",
"ref":"fedora/f26/x86_64/atomic-host"}

'message-content' is not serializable

Downstream CI listens for this fed message from upstream in our pipeline:
org.centos.prod.ci.pipeline.image.test.smoke.complete

One of the fields in the message we receive, 'message-content', has a dash in it and thus is not serializable when we try to parse it and load it as an env variable using the same sort of logic upstream CI does.

It would be good to replace that dash with an underscore or similar.

This is not really a field we use but others could be hit by that same thing.

I currently work around this by altering how we parse our fields and replace the dashes with underscores, but it would be better to have the actual message fixed.

RAW Message body from a org.centos.prod.ci.pipeline.image.test.smoke.complete we received:

{"CI_TYPE":"custom","build_id":"155","status":"SUCCESS","username":"fedora-atomic","compose_url":"http://artifacts.ci.centos.org/artifacts/fedora-atomic/f26/ostree","rev":"78666125c0f3679f95a1867353ae43ad4b498475","message-content":"","build_url":"https://jenkins-continuous-infra.apps.ci.centos.org/job/ci-pipeline-f26/155/","namespace":"rpms","CI_NAME":"ci-pipeline-f26","image_name":"fedora-atomic-26.229-d7aeb22689b8c7a.qcow2","repo":"kubernetes","compose_rev":"d7aeb22689b8c7a97b0d8ffda64cac15587d869c8052031c3e8d7dd31e528ef1","topic":"org.centos.prod.ci.pipeline.image.test.smoke.complete","image_url":"http://artifacts.ci.centos.org/artifacts/fedora-atomic/f26/images/fedora-atomic-26.229-d7aeb22689b8c7a.qcow2","branch":"f26","type":"qcow2","test_guidance":"''","ref":"fedora/f26/x86_64/atomic-host"}

I talked to Ari about this last week and we both agree this is pretty low pri/impact

The original_spec_nvr field includes the branch name not the dist-tag

I was looking at how to display the results of the Atomic CI pipeline in bodhi using criu-3.3-5.f26 as an example since it exists in resultsdb: https://taskotron.fedoraproject.org/resultsdb/results/17377652

I had quite some problem creating the update in my local bodhi update because I was using criu-3.3-5.f26 taken from resultsdb above. But this NVR isn't valid, the proper one is: criu-3.3-5.fc26 (.fc26) which is what bodhi knows and queries resultsdb with.

Could we adjust the original_spec_nvr so it is the dist-tag and not the branch name?

virt-customize: incorrectly not installing conflicting RPM

The ignition package has recently separated out a subpackage called ignition-validate. ignition-validate can run stand alone but is only validates ignition config files from ignition > 0.31.0-3 so there is a conflict with ignition < 0.31.0-3. I believe this is the proper use of conflicts vs requires for this package.

During the setup of the test environment, even though ignition is from koji is > 0.31.0-3 , it is not being installed [1]. I think the problem is in virt-customize.sh [0] where it queries for a conflict but only looks at the package name to determine to install it or not. virt-customize.sh should look at the entire query output and determine if the package to be installed is actually at a version that conflicts.

[0]

ci-pipeline/config/Dockerfiles/cloud-image-compose/virt-customize.sh

Lines 112 to 114 in 0b06c15

conflict=$(repoquery -q --disablerepo=\* --enablerepo=${package} --repofrompath=${package},${rpm_repo} --conflict $pkg | awk '{print$1}')
found_conflict=0
if [ ! -z "${conflict}" ] && [ ! -z "${RPM_LIST}" ]; then

[1] snippet from pipeline.log


[2019-03-22T21:57:05.391Z] + '[' rpms '!=' tests ']'
[2019-03-22T21:57:05.391Z] ++ repoquery -q '--disablerepo=*' --enablerepo=ignition --repofrompath=ignition,/workDir/workspace/fedora-rawhide-pr-pipeline/ignition_repo --all '--qf=%{ARCH}:%{NAME}'
[2019-03-22T21:57:05.391Z] ++ sed -e '/^src:/d;/-debug\(info\|source\)$/d;s/.\+://'
[2019-03-22T21:57:05.391Z] ++ sort -u
[2019-03-22T21:57:05.647Z] BDB2053 Freeing read locks for locker 0xc0: 105/140149436913792
[2019-03-22T21:57:05.647Z] BDB2053 Freeing read locks for locker 0xc2: 105/140149436913792
[2019-03-22T21:57:05.647Z] BDB2053 Freeing read locks for locker 0xc3: 105/140149436913792
[2019-03-22T21:57:05.647Z] BDB2053 Freeing read locks for locker 0xc4: 105/140149436913792
[2019-03-22T21:57:05.647Z] + for pkg in '$(repoquery -q --disablerepo=\* --enablerepo=${package} --repofrompath=${package},${rpm_repo} --all --qf="%{ARCH}:%{NAME}" | sed -e "/^src:/d;/-debug\(info\|source\)\$/d;s/.\+://" | sort -u)'
[2019-03-22T21:57:05.647Z] ++ repoquery -q '--disablerepo=*' --enablerepo=ignition --repofrompath=ignition,/workDir/workspace/fedora-rawhide-pr-pipeline/ignition_repo --conflict ignition
[2019-03-22T21:57:05.647Z] ++ awk '{print$1}'
[2019-03-22T21:57:05.903Z] + conflict=
[2019-03-22T21:57:05.903Z] + found_conflict=0
[2019-03-22T21:57:05.903Z] + '[' '!' -z '' ']'
[2019-03-22T21:57:05.903Z] + RPM_LIST=' ignition'
[2019-03-22T21:57:05.903Z] + for pkg in '$(repoquery -q --disablerepo=\* --enablerepo=${package} --repofrompath=${package},${rpm_repo} --all --qf="%{ARCH}:%{NAME}" | sed -e "/^src:/d;/-debug\(info\|source\)\$/d;s/.\+://" | sort -u)'
[2019-03-22T21:57:05.903Z] ++ repoquery -q '--disablerepo=*' --enablerepo=ignition --repofrompath=ignition,/workDir/workspace/fedora-rawhide-pr-pipeline/ignition_repo --conflict ignition-validate
[2019-03-22T21:57:05.903Z] ++ awk '{print$1}'
[2019-03-22T21:57:06.159Z] + conflict=ignition
[2019-03-22T21:57:06.159Z] + found_conflict=0
[2019-03-22T21:57:06.159Z] + '[' '!' -z ignition ']'
[2019-03-22T21:57:06.159Z] + '[' '!' -z ' ignition' ']'
[2019-03-22T21:57:06.159Z] + for rpm_pkg in '${RPM_LIST}'
[2019-03-22T21:57:06.159Z] + '[' ignition == ignition ']'
[2019-03-22T21:57:06.159Z] + found_conflict=1
[2019-03-22T21:57:06.159Z] + continue
[2019-03-22T21:57:06.159Z] + '[' 1 ']'
[2019-03-22T21:57:06.159Z] + echo 'INFO: will not install ignition-validate as it conflicts with ignition.'
[2019-03-22T21:57:06.159Z] INFO: will not install ignition-validate as it conflicts with ignition.

ci.pipeline.complete does not include original_spec_nvr when failing

Good Morning,

I was going through some of the ci.pipeline.complete messages this morning and realized that when the message is about a failure, it does not include the original_spec_nvr field.
Which means we would not be able to link that message to a bodhi udpate.

Some example:

Improve test results reporting

There has been a discussion about how to improve reporting results in the Standard Test Interface so that it is more easy to distinguish test failure from infrastructure error:

Let's update the pipeline according to the agreed result: The main difference is that ansible-playbook should return non-zero status only upon infrastructure errors while tests failures can be detected from the newly defined file results.yml which will have the following format:

results:
- {result: pass, test: shell/smoke}
- {result: fail, test: shell/login}
- {result: error, test: shell/func}

Extended version with links to logs:

results:
- {result: pass, test: shell/smoke, log: /tmp/artifacts/PASS_shell-smoke.log}
- {result: fail, test: shell/login, log: /tmp/artifacts/PASS_shell-login.log}
- {result: error, test: shell/func, log: /tmp/artifacts/FAIL_shell-func.log}

The expected behaviour of the pipeline is this:

  • Report error when ansible returns non-zero (infrastructure error)
  • Report error if error appears in results.yml files
  • Report test failure if fail appears in results.yml files
  • Report pass otherwise

This issue is blocked by implementation of the new functionality in Standard Test Roles which is tracked here:

Mechanism to re-trigger a test run? (The idea and the specifications)

Sometime the CI pipeline fails due to a bug in the pipeline, an infrastructure issue or an outage in one of the system it depends (like the small outage that dist-git had in Fedora 2 days ago).

When this happens, if gating is enabled, the update will end up being blocked in bodhi because its tests failed. Currently the only way around this would be to "waive" the test results in waiverdb. However, this should be a rel-eng only action and is going a little bit against the idea of CI (since we would be waiving the results not because the tests failed but because we failed to run them).

The better solution, would be to have a way to re-trigger a pipeline run.

There is already a discussion that started on fedora-infra/bodhi#1779 to add a 'Re-run test' button to bodhi for taskotron tests. I believe we should try to come up with a solution that satisfies both taskotron and the Atomic CI pipeline.

My proposal on that PR was to have a dedicated service that would receive API calls and emit fedmsg message with the information needed to re-run the tests.
(Having it a separate service would allow integrating it in multiple places, such as bodhi and pagure)

What do you think of this idea?

Could you describe what information you would need in a fedmsg message to re-trigger a run? (Keeping in mind that bodhi does not know the commit hash corresponding to a build, so we may only be able to provide the combo repository/branch).

@AdamWill Could you also described what information you would like to have to re-trigger a test in taskotron?

Thanks :)

Which host is used to run the all package pipeline

Looking at the reason of the last failure (this morning): https://jenkins-continuous-infra.apps.ci.centos.org/view/Fedora%20All%20Packages%20Pipeline/job/upstream-fedora-f27-pipeline/104/console

It seems it fails with:

06:42:56 + git checkout -b test_branch
06:42:56 Switched to a new branch 'test_branch'
06:42:56 ++ rpm -q --define 'dist .fc27' --queryformat '%{name}-%{version}-%{release}\n' --specfile php-cs-fixer.spec
06:42:56 ++ head -n 1
06:42:56 error: line 48: Unknown rich dependency op 'with': BuildRequires:  (php-composer(composer/semver)               >= 1.4 with php-composer(composer/semver)               < 2)
06:42:56 error: query of specfile php-cs-fixer.spec failed, can't parse

The same command runs fine for me locally (on a F27 box)

$ rpm -q --define 'dist .fc27' --queryformat '%{name}-%{version}-%{release}\n' --specfile php-cs-fixer.spec
php-cs-fixer-2.11.0-1.fc27

Any idea what could lead to this? Could it be a difference in the underlying OS (F27 vs one that doesn't know reach dependencies?)

How long are jobs kept? Where to point users?

I opened #287 3 days ago and it points to a job: https://jenkins-continuous-infra.apps.ci.centos.org/job/continuous-infra-ci-pipeline-f26/30/ that is now returning a 404.

This is, however, the link that is present in the messages and sent to the packagers, so this leads to two questions:

  • how long are the jobs kept?
  • if this url isn't stable, do we have a better one to send the packagers to? So they can see, 3 hours, 2 weeks, a month later why a build didn't pass CI

Thanks

use koji repos for testing

Right now we use just fedora+updates(+updates-testing)? for released fedora and just rawhide for rawhide.

This means that if you update multiple packages simultaneously, your test pipeline will fail because your dependency is not yet in compose.

Failed job but no corresponding message?

This night I got two notifications on irc:

00:43:33 fedora-notifstg | Commit 876dc24d of package rpms/selinux-policy is being built in the CI pipeline on branch f26 https://jenkins-continuous-infra.apps.ci.centos.org/job/continuous-infra-ci-pipeline-f26/30/
01:30:30 fedora-notifstg | Commit 876dc24d of package rpms/selinux-policy is being built in the CI pipeline on branch f26 https://jenkins-continuous-infra.apps.ci.centos.org/job/continuous-infra-ci-pipeline-f26/30/

Since I expected to get more, I went to the job on jenkins: https://jenkins-continuous-infra.apps.ci.centos.org/job/continuous-infra-ci-pipeline-f26/30/

This raised two questions:

  • Just under the red dot next to the title Build Build#: 30... is written: SUCCESS but if I hover over the dot it says: Failed.

In my (limited) experience with jenkins, red is failed and blue succeeded, so I will assume the job failed, but this SUCCESS is a little confusing, I don't know if we can do something about it.

Thanks :)

Message *.running without their corresponding .complete

Looking at the recent entries in https://apps.stg.fedoraproject.org/datagrepper/raw?category=ci
I see two compose.running and images.running without their corresponding .complete messages.

Looking at https://apps.stg.fedoraproject.org/datagrepper/id?id=2017-c1b3ca26-c1f3-454a-88ef-7a41f379be46&is_raw=true&size=extra-large (compose.running) and https://apps.stg.fedoraproject.org/datagrepper/id?id=2017-40ca49ec-579d-4204-a4a0-054861a35f56&is_raw=true&size=extra-large (image.running) the repo, commit hash and branch are the same, so I expect the image was created from that compose, but either the .complete message didn't get sent or it didn't reach datagrepper, either one being a bummer.

Could we double-check if something is going on?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.