sncf-connect-tech / hesperides-jenkins-lib

Shared lib for Jenkins pipelines to interact with Hesperides

This is a Groovy shared lib for Jenkins 2 pipelines, but also useable as a Groovy command-line script.

It lets you interact with Hesperides. It provides some high level features over the bare HTTP REST API:

• setPlatformModulesVersion: change the version of all modules on a plateform
• updateProperties: change instance/module/global properties on a given Hesperides platform, from a JSON file listing deltas
• upsertFromDescriptor: upsert (create or update) one or several workingcopy module(s) and their templates from a file descriptor

This project is actively maintained. A detailed releases changelog is available here: CHANGELOG.md

A documentation for the exposed Groovy functions is available here: https://voyages-sncf-technologies.github.io/hesperides-jenkins-lib/

Summary

• Installation
• Usage
  • Jenkins pipeline
  • CLI script for the standard Groovy interpreter
• Contributing / Development

Installation

In "Manage Jenkins > Configure System" aka /jenkins/configure, adds the git@... URL to this repo in Global Pipeline Libraries, and ticks the "load implicitly" checkbox. You can use either the master branch to use the latest version, or specify a tag to ensure more stability.

You will also need to install the http_request Jenkins plugin from its .hpi.

Usage

Jenkins pipeline

cf. examples/Jenkinsfile & vars/*.txt documentation files for examples.

Note: to check your Jenkinsfiles syntax, use a linter ! cf. https://github.com/Lucas-C/pre-commit-hooks#other-useful-local-hooks

CLI script for the standard Groovy interpreter

cf. examples/createPlatform.groovy

Contributing / Development

cf. CONTRIBUTING.md

hesperides-jenkins-lib's Issues

Include in the documentation a mention of optional arguments

Currently they are only mentioned as inline comments in the code:
https://github.com/voyages-sncf-technologies/hesperides-jenkins-lib/blob/master/src/com/vsct/dt/hesperides/jenkins/pipelines/Hesperides.groovy#L41

We should mention them in hesperides.txt, either by migrating them there, or by automating their insertion in hesperides.txt (by extrating each corresponding def line from the Hesperides class ?)

Add new method to POST request to /applications/{application_name}/platforms/{platform_name}/properties

Currently the jenkins lib provide a method to GET platform properties but not update it.
I'm working on a PR to add one.

Disable error for no matching modules in selectModules

La problématique est que sur certaines plateformes, tous les modules ne sont pas installés.

Le job jenkins utilise aujourd'hui la fonction setPlatformModuleVersion, de façon itérative sur tous les modules de la note de livraison.
`

                    modules = deliveryNote.getDeliveryNoteHesperidesAllModulesDatas(trigram: jsonConfig.trigram, deliveryNote: deliveryNoteChosen)
                    echo "Hesperides Modules from delivery note: => $modules"

                    modules.each {
                        def String hesperideModuleName = it.value.hesperidesModuleName
                        def String hesperideModuleVersion = it.value.hesperidesModuleVersion

                        echo "Switch Hesperides module =>  App: $jsonConfig.trigram, Pltf: $deployPlatform, module: $hesperideModuleName, version: $hesperideModuleVersion"
                        hesperides.setPlatformModuleVersion(
                                auth: hesperidesCredentials,
                                app: jsonConfig.trigram,
                                platform: deployPlatform,
                                newVersion: hesperideModuleVersion,
                                moduleName: hesperideModuleName,
                                isWorkingcopy: workingCopy
                        )
                    }

`
Dans la librairie hesperides, setPlatformModuleVersion appelle ensuite selectModules, et tout cela termine en échec si le module en question n'existe pas sur la plateforme.

Est-il possible de faire renvoyer à selectModules une liste vide et un warning si le module n'est pas trouvé ?
En effet, même si certains composants sont livrés ensemble car ils ont le même cycle de vie (développement), ils ne sont pas tous susceptible d'être installé sur chaque plateforme.

Des exemple VSCT sur HUP , (avec le MOCK et les Batches).

Peut-être une autre solution en amont, dans le JenkinsFile serait préférable, mais là je ne vois pas trop sans faire du code spécifique à chaque composant.

Bug lorsque le logicGroupPath fourni à putModuleOnPlatform est préfixé d'un #

Dans ce cas, la lib envoie à l'API un properties_path préfixé de deux dièses

Ajouter le version_id et les cookies dans les logs lors des appels à Hespérides

Failing test in CircleCI

HesperidesIntegrationSpec > Can download all files of an instance STANDARD_ERROR
    POST http://hesperides:8080/rest/modules/module/2.0.0.0/workingcopy/templates
    {
        "name": "titi",
        "filename": "titi",
        "location": "/etc",
        "content": "iam=titi",
        "version_id": -1,
        "rights": {
            "user": {
            
            },
            "group": {
            
            }
        }
    }
    Content-Type: null
    Accept: JSON
    POST http://hesperides:8080/rest/modules/module/2.0.0.0/workingcopy/templates
    {
        "name": "toto",
        "filename": "toto",
        "location": "/etc",
        "content": "iam=toto",
        "version_id": -1,
        "rights": {
            "user": {
            
            },
            "group": {
            
            }
        }
    }
    Content-Type: null
    Accept: JSON
    GET http://hesperides:8080/rest/applications/app/platforms/platform
    Content-Type: null
    Accept: JSON
    GET http://hesperides:8080/rest/files/applications/app/platforms/platform/%23GROUP%23TECHNO/module/2.0.0.0/instances/instance/?isWorkingCopy=false
    Content-Type: null
    Accept: JSON
    {
        "status": 404,
        "exception": "com.vsct.dt.hesperides.exception.runtime.MissingResourceException",
        "message": "There is no module module/2.0.0.0/Release",
        "stacktrace": "com.vsct.dt.hesperides.exception.runtime.MissingResourceException: There is no module module/2.0.0.0/Release\n\tat com.vsct.dt.hesperides.files.Files.lambda$getLocations$103(Files.java:108)\n\tat java.util.Optional.orElseThrow(Optional.java:290)\n\tat com.vsct.dt.hesperides.files.Files.getLocations(Files.java:108)\n\tat com.vsct.dt.hesperides.resources.HesperidesFilesResource.getFilesList(HesperidesFilesResource.java:82)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)\n\tat com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)\n\tat com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)\n\tat com.codahale.metrics.jersey.InstrumentedResourceMethodDispatchProvider$TimedRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:30)\n\tat io.dropwizard.jersey.guava.OptionalResourceMethodDispatchAdapter$OptionalRequestDispatcher.dispatch(OptionalResourceMethodDispatchAdapter.java:37)\n\tat com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)\n\tat com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)\n\tat com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)\n\tat com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)\n\tat com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)\n\tat com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)\n\tat com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:848)\n\tat io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)\n\tat org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)\n\tat org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:348)\n\tat io.dropwizard.jetty.BiDiGzipFilter.doFilter(BiDiGzipFilter.java:127)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)\n\tat io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:29)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:173)\n\tat io.dropwizard.jetty.ContextRoutingHandler.handle(ContextRoutingHandler.java:38)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:162)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:446)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)\n\tat java.lang.Thread.run(Thread.java:745)\n"
    }

HesperidesIntegrationSpec > Can download all files of an instance FAILED
    com.vsct.dt.hesperides.jenkins.pipelines.http.HttpException: Not Found : 404
        at com.vsct.dt.hesperides.jenkins.pipelines.http.HTTPBuilderRequester.performRequest_closure1$_closure3(HTTPBuilderRequester.groovy:66)
        at groovy.lang.Closure.call(Closure.java:414)
        at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:503)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)
        at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
        at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:434)
        at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:383)
        at com.vsct.dt.hesperides.jenkins.pipelines.http.HTTPBuilderRequester.performRequest(HTTPBuilderRequester.groovy:49)
        at com.vsct.dt.hesperides.jenkins.pipelines.Hesperides.httpRequest(Hesperides.groovy:633)
        at com.vsct.dt.hesperides.jenkins.pipelines.Hesperides.getInstanceFiles(Hesperides.groovy:586)
        at HesperidesIntegrationSpec.Can download all files of an instance(HesperidesIntegrationSpec.groovy:417)

Ajouter une fonction de nettoyage des propriétés supprimées d'un module ou d'une plateforme

En attente de : sncf-connect-tech/hesperides#803

Iterables propertis send old + new values

Lorsque je veux mettre des props itérables, les nouvelles + les anciennes valeurs sont renseignées dans la requête de MAJ à Hesperides. Mon changeset :

"nf-geoBuilder": {
    "els_user" : "...",
    "els_password" : "...",
    "iterable_properties": {
      "elasticsearch.indexes": [
        {
          "alias_name": "geo_{{hesperides.platform.name}}",
          "els_hosts": "sermaglia,barghe,monte",
          "els_port": 52300
        },
        {
          "alias_name": "geo_INT6",
          "els_hosts": "sermaglia,barghe,monte",
          "els_port": 52300
        }
      ]
    }
  }

La requête qui est envoyée à Hesperides contient les valeurs suivantes dans les iterables props :

{
    "iterable_properties": [
        {
            "iterable_valorisation_items": [
                {
                    "values": [
                        {
                            "name": "els_port",
                            "value": "52300"
                        },
                        {
                            "name": "alias_name",
                            "value": "catalog_{{hesperides.platform.name}}"
                        },
                        {
                            "name": "els_hosts",
                            "value": "zangarella,padiglione,bologne"
                        }
                    ],
                    "title": ""
                },
                {
                    "values": [
                        {
                            "name": "els_port",
                            "value": "52300"
                        },
                        {
                            "name": "alias_name",
                            "value": "catalog_INT6"
                        },
                        {
                            "name": "els_hosts",
                            "value": "zangarella,padiglione,bologne"
                        }
                    ],
                    "title": ""
                }
            ],
            "name": "elasticsearch.indexes"
        },
        {
            "name": "elasticsearch.indexes",
            "iterable_valorisation_items": [
                {
                    "title": "not used",
                    "values": [
                        {
                            "name": "els_port",
                            "value": 52300
                        },
                        {
                            "name": "alias_name",
                            "value": "catalog_{{hesperides.platform.name}}"
                        },
                        {
                            "name": "els_hosts",
                            "value": "sermaglia,barghe,monte"
                        }
                    ]
                },
                {
                    "title": "not used",
                    "values": [
                        {
                            "name": "els_port",
                            "value": 52300
                        },
                        {
                            "name": "alias_name",
                            "value": "catalog_INT6"
                        },
                        {
                            "name": "els_hosts",
                            "value": "sermaglia,barghe,monte"
                        }
                    ]
                }
            ]
        }
    ]
}

Le serveur ne semble prendre en compte que le premier item du tableau car mes props itérables ne sont pas mises à jour.

Use a newer version of hesperides in docker-compose.yml

We currently use a one-year old Docker image tagged demo-chaos, cf. https://github.com/voyages-sncf-technologies/hesperides-jenkins-lib/blob/master/docker-compose.yml#L9

We need to ensure everything still works fine with the new, currently in development, version of Hesperides. E.g. by creating a dedicated branch in this repo that uses the hesperides/hesperides:develop Docker image

Consistency issue in the isWorkingCopy attribute

The isWorkingCopy attribute has case inconsistency across the various methods, it's sometimes written isWorkingcopy (no uppercase C).

While workingCopy (no verb) should be preferred, I'm currently working on aligning expected parameters on isWorkingCopy (camel case), while being backward compatible.

Bug de MAJ des propriétés d'instance

Fichier d'input :

{
  "moduleName#INSTANCE_NAME": {
    "hostname": "toto"
  }
}

Log observé :

------------ module name for specific instance: moduleName#INSTANCE_NAME
[instance=CUSWRBESI11] NEW property hostname: toto
PUT .../platforms?application_name=XXX&copyPropertiesForUpgradedModules=true
{
    "modules": [
        {
            "path": "#FOO#BAR",
            "working_copy": true,
            "instances": [
                {
                    "key_values": [
                        {
                            "name": "hostname",
                            "value": ""
                        },
                        {
                            "name": "hesperides.instance.name",
                            "value": ""
                        }
                    ],
                    "name": "INSTANCE_NAME"
                }
            ],
            "key_values": [
                {
                    "name": "hostname",
                    "value": "toto"
                }
            ],
            "name": "moduleName",
            "properties_path": "...",
            "id": 10,
            "version": "1.2.3-SNAPSHOT"
        },

Bug de MAJ des propriétés itératives avec un path: - 400 Bad Request

Bonjour,
lorsqu'on injecte les properties via la lib groovy, 3 comportements ont été vu sur les properties itératives :
1 - quand on push les properties avec un json du type :

{
  "houston-event-stream-api-jar": {
    "search.courses.responseMaxSize": "100"
  }
}

cela supprime les itérative, on est obligé de les indiquer a chaque json de diff de properties.

2 - quand on push les properties avec une notion de path :

{
  "path:#HCO#JAR#houston-connector-pao-jar": {
    "streams.ticket-mark.kafka.destination-topic": "ticket-marks-bom",
    "streams.ticket-mark.throttle.max-request-per-seconds": "20"
  }
}

Là il y a conservation des itératives, ce qui est un comportement plutot attendu.

3 - lorsqu'on tente de push des properties itérative avec la notion de path :

{
  "path:#HCO#JAR#houston-connector-pao-jar": {
    "streams.ticket-mark.accept-delay-days-to-midnight": "1",
    "iterable_properties": {
     "pao.kafka.consumer.kafka-clients.bootstraps.servers": [
        {
          "pao.kafka.consumer.kafka-clients.bootstrap.server.id": "int-2",
          "pao.kafka.consumer.kafka-clients.bootstrap.server.value": "localhost:50000"
        }
      ]
    }
  }
}

Là la lib groovy nous balance qu'il n'arrive pas a déserializer.

Pour le moment pour le pas pourrir les iterative je passe uniquement par le path.