Coder Social home page Coder Social logo

jenkins-formula's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jenkins-formula's Issues

Provide master-slave states

Hi,

Here at PeopleDoc we built a salt formula for deploying master/slave jenkins. Our target is to fully provision jenkins through salt. We open sourced the code at https://github.com/novafloss/ci-formula . What do you thing of merging ci-formula in jenkins-formula to address more use case.

In the end we would have jenkins.master and jenkins.slave entry point. We could also add jenkins_job, jenkins_view and jenkins_plugin states, but this require jenkins-cli.

If you are ok, i can do some pull requests to progressively merge interesting feature in official jenkins-formula.

Regards,
Γ‰tienne

Put all macros in a different file and import them as needed

Just a slight suggestion and I also wish to ask for your opinion about this:

There are a number of macros in the sls files. How about we put them into a single file called macros and comment each macros what it does. Then import them with a single line into any states files if they are needed.

A better approach may yet be to separate the macro files based on topic:

macro-cli
macro-jenkins
macro-etc...

This way the user knows exactly which macro is written in which file and if someone just need to update one, they can do so without messing around with the other files.

This will, however, only add more benefit if this project becomes larger and we have more macros to maintain. To make it future proof, I do suggest we start now with this.

[BUG] RHEL repo gpg error due to incorrect key

----------
          ID: jenkins
    Function: pkg.installed
      Result: False
     Comment: Error occurred installing package(s). Additional info follows:

              errors:
                  - Running scope as unit: run-ra5684b079467415dbb0c6ac1ff786386.scope
                    Last metadata expiration check: 0:00:02 ago on Mon Mar 25 19:48:31 2024.
                    Dependencies resolved.
                    ================================================================================
                     Package          Architecture    Version                Repository        Size
                    ================================================================================
                    Installing:
                     jenkins          noarch          2.440.2-1.1            jenkins           82 M

                    Transaction Summary
                    ================================================================================
                    Install  1 Package

                    Total download size: 82 M
                    Installed size: 83 M
                    Downloading Packages:
                    jenkins-2.440.2-1.1.noarch.rpm                   45 MB/s |  82 MB     00:01
                    --------------------------------------------------------------------------------
                    Total                                            44 MB/s |  82 MB     00:01
                    Jenkins upstream package repository              40 kB/s | 3.1 kB     00:00
                    Importing GPG key 0x45F2C3D5:
                     Userid     : "Jenkins Project <[email protected]>"
                     Fingerprint: 62A9 756B FD78 0C37 7CF2 4BA8 FCEF 32E7 45F2 C3D5
                     From       : https://pkg.jenkins.io/redhat-stable/jenkins.io.key
                    Key imported successfully
                    Import of key(s) didn't help, wrong key(s)?
                    Public key for jenkins-2.440.2-1.1.noarch.rpm is not installed. Failing package is: jenkins-2.440.2-1.1.noarch
                     GPG Keys are configured as: https://pkg.jenkins.io/redhat-stable/jenkins.io.key
                    The downloaded packages were saved in cache until the next successful transaction.
                    You can remove cached packages by executing 'dnf clean packages'.
                    Error: GPG check FAILED
     Started: 19:48:26.886320
    Duration: 9909.231 ms
     Changes:
              ----------
              gpg-pubkey.(none):
                  ----------
                  new:
                      3228467c-613798eb,350d275d-6279464b,45f2c3d5-5e81efb9,621e9f35-58adea78,73d76b3f-63dace60,7bd9bf62-5762b5f8
                  old:
                      3228467c-613798eb,350d275d-6279464b,621e9f35-58adea78,73d76b3f-63dace60,7bd9bf62-5762b5f8

Formula commit hash / release tag

- jenkins-formula:
          rev: 'da763dd'    # 2024.03 - git rev-parse --short HEAD, tags are not available for the repo

Problem line is https://github.com/saltstack-formulas/jenkins-formula/blob/da763dd88f0b7871ff01272b3a894e0bebf680f7/jenkins/init.sls#L46C7-L46C76

Fix

- gpgkey: https://pkg.jenkins.io/redhat{{ repo_suffix }}/jenkins.io.key
should be replaced with line
- gpgkey: https://pkg.jenkins.io/redhat{{ repo_suffix }}/jenkins.io-2023.key

or move key name to defaults so it could be changed via lookup

Unable to install plugins with default debian config

I have tried to install plugins with default config.xml file on debian 8, but jenkins cli doesn't work due to error: No X-Jenkins-CLI2-Port among.
I have fix it in config.xml - changed value from default <slaveAgentPort>-1</slaveAgentPort> to 0 (random port) a restart jenkins service before jenkins_listening
Tested on jenkins version 2.7/2.8 instaled by this formula.
Does someone have same problem with plugins/cli or some better solution?

Question: Reason for Jinja import after YAML include

Just curious if it makes any difference to put the jinja imports before the YAML include found in jobs.sls for example.

What is the best practice if if it does not make a difference, which one should we stick to?

Formula assumes Java installed

Hi,

When running this formula on a very clean box, such that it minimal install doesn't include a Java version. The formula breaks in an somewhat goofy way: Jenkins has dependency on Java to run. Perhaps include a test that Java is installed and at some reasonable minimal version ?
or a doc change that suggests including: https://github.com/saltstack-formulas/java-formula

Plugins won't install if security is enabled on jenkins

There should be a way of providing jenkins credentials in order to use jenkins-cli. Jenkins-cli uses rest calls to install plugins. jenkins.plugins state will basically break as soon as you setup security on jenkins.

e.g. I would like to use jenkins github oauth plugin for authentication: https://wiki.jenkins-ci.org/display/JENKINS/Github+OAuth+Plugin

There should be a way of passing github oauth token for connections between jenkins-cli and local jenkins instance. All credentials should be provided in pillar.

avoid 2.0 setup wizard,make secure, and disable anonymous user

With newer versions of Jenkins, now have setup wizard to contend with. Found a decent way to override that by adding the following to map.jinja (java args line 11):
'java_args': '-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false',

seems there are other groovy scripts to disable the anonymous user and turn on security that we could somehow include ?

see:
https://groups.google.com/forum/#!msg/jenkinsci-users/Pb4QZVc2-f0/ywKqZVf9MgAJ

Installing Jenkins Plugins

Here is a simple hack to install plugins, maybe someone likes it.
It is working now, needs some settings to be installed.
you will also need to restore /var/lib/jenkins/jenkins_settings/installedPlugins.xml and then
you will have a bunch of things like /var/lib/jenkins/*.xml that need to be setup.

{% set plugins = ("analysis-collector","analysis-core","ansicolor","ant","artifactdeployer","artifactory","batch-task","build-metrics","build-publisher","checkstyle","chucknorris","cobertura","conditional-buildstep","configurationslicing","confluence-publisher","copyartifact","credentials","cvs","dashboard-view","deploy","disk-usage","email-ext","envinject","extended-choice-parameter","external-monitor-job","extra-columns","fail-the-build-plugin","git-client","git-server","git","global-build-stats","greenballs","groovy-postbuild","groovy","htmlpublisher","javadoc","jenkinswalldisplay","jira","jobConfigHistory","jquery","ldap","log-parser","mailer","mask-passwords","maven-info","maven-plugin","multiple-scms","nested-view","nodelabelparameter","pam-auth","parameterized-trigger","pmd","promoted-builds-simple","promoted-builds","publish-over-ssh","python","run-condition","scm-api","scm-sync-configuration","scp","scriptler","sectioned-view","selenium-aes","selenium","seleniumhq","shelve-project-plugin","sidebar-link","sounds","ssh-agent","ssh-credentials","ssh-slaves","ssh","subversion","svn-tag","svnpublisher","testng-plugin","text-finder","thinBackup","token-macro","translation","view-job-filters","ws-cleanup") %}

{% for plugin in plugins %}
jenkins_install_plugin_{{plugin}}_1:
  cmd.run:
    - name : curl -k -L https://updates.jenkins-ci.org/latest/{{plugin}}.hpi -o /var/lib/jenkins/plugins/{{plugin}}.hpi
    - onlyif: 'test ! -f /var/lib/jenkins/plugins/{{plugin}}.hpi'

jenkins_install_plugin_{{plugin}}_2:
  file.managed:
    - {name: "/var/lib/jenkins/plugins/{{plugin}}.hpi"}
    - {user: 'jenkins'}
    - {group: 'jenkins'}
    - {replace : False }
    - require:
      - cmd : jenkins_install_plugin_{{plugin}}_1

jenkins_install_plugin_{{plugin}}_3:
  archive.extracted:
    - name: "/var/lib/jenkins/plugins/{{plugin}}/"
    - source: "/var/lib/jenkins/plugins/{{plugin}}.hpi"
    - archive_format: zip
    - user: jenkins
    - group: jenkins
    - onlyif: 'test ! -f /var/lib/jenkins/plugins/{{plugin}}'
    - require:
      - file : jenkins_install_plugin_{{plugin}}_2

{% endfor %}

kitchen converge is broken...

kitchen converge is not working. First thing was it wanted me to create the vendor_path, mkdir vendor

No it can't find dependencies and is failing, I'm fairly new to salt but have used Kitchen (with Chef) a bunch. Any advice about how to best tackle this. I know kitchen-salt has dependencies but I can't get them to work:

saltstack/kitchen-salt#130

Allow setting server name

Currently, the server_name nginx configuration variable for jenkins is commented.

It would be great if we could override it, so that other sites can live along with jenkins on this nginx configuration.

nginx States are hard coded and not friendly for Centos/Redhat

Problem:

When I try to use jenkins.nginx to install nginx config for jenkins, it does not work on Centos 7 since sites-available and enabled are not the correct folders.

Solution:

I used this custom state instead:

# Install nginx, run nginx only if installation is successful

nginx:
  pkg.installed: []

/etc/nginx/default.d/jenkins.conf:
  file.managed:
    - source: salt://nginx/files/jenkins.conf
    - user: nginx
    - group: nginx
    - require:
      - pkg: nginx

/etc/nginx/nginx.conf:
  file.managed:
    - source: salt://nginx/files/nginx.conf
    - user: nginx
    - group: nginx
    - require:
      - pkg: nginx

Run nginx when everything is ready:
  service.running:
    - name: nginx
    - enable: True
    - reload: True
    - require:
      - pkg: nginx
    - watch:
      - file: /etc/nginx/default.d/jenkins.conf
      - file: /etc/nginx/nginx.conf

The files are placed at the usual directories within the nginx directory I created.

Proposed Solution:

To get rid of the hard coded version and make further centos and redhat friendly version available through map.jinja and pillar. Make the source file pillar configurable and the map.jinja include variables for the final path of clients depending on the linux distribution.

Provide platform-specific default settings

This for now is mainly to resolve the netcat-related issues that have come up around #47 - as netcat (the package) is named differently and (the binary) has different flags it looks like a good solution to me to add platform-specific settings to map.jinja - i'll follow up with a PR asap.

Include flexibility to install either LTS or Dev Jenkins Version

Problem:

When using jenkins to install Jenkins I always get the development version but not the stable LTS. I wish to choose which one to install.

Solution:

I used the following to extend your formula:

extend:
  jenkins:
    {% if grains['os_family'] in ['RedHat', 'Debian'] %}
    pkgrepo.managed:
      - humanname: Jenkins upstream package repository
      {% if grains['os_family'] == 'RedHat' %}
      # Upgrade to Jenkins 2.7 LTS
      - baseurl: http://pkg.jenkins-ci.org/redhat-stable
      - gpgkey: http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
      {% elif grains['os_family'] == 'Debian' %}
      - file: {{jenkins.deb_apt_source}}
      - name: deb http://pkg.jenkins-ci.org/debian binary/
      - key_url: http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key
      {% endif %}
      - require_in:
        - pkg: jenkins
    {% endif %}
    pkg.installed:
      - pkgs: {{ jenkins.pkgs|json }}
    service.running:
      - enable: True
      - watch:
        - pkg: jenkins

Proposed Solution:

I wish to propose that you add a variable for the repo distro, i think it is just an one liner. Include this in the map.jinja and pillar so users can use pillar to choose whether to install LTS or the dev version of Jenkins.

If I have the time, I will definitely make a PR. Let me know if you have the time to do it yourself soon, otherwise, I can try to make some time on my side asap. Just wish to know your time schedule so I don't work on the same thing as you do.

Jobs do not update when XML file is edited

We recently started using jenkins-formula for managing all aspects of Jenkins.

When an XML job def is edited, the new file is copied to the server, but the job is not updated and therefore the new configuration is not used.

Steps to reproduce:

  • Create a new job XML file
  • Run state.apply
  • Note that the new job appears πŸŽ‰
  • Edit the job XML file
  • Run state.apply
  • Note that the job config is not updated 😞

I have an incoming PR to fix this.

netcat-openbsd Hard to Get Installed on Centos 7

First off, many thanks for putting the effort to create this formula, I am using it at my work and it is a bliss.

The problem:

When I used jenkins.plugins to install the plugins for Jenkins, jenkins_listening returns false since netcat-openbsd is not installed on Centos 7. I tried to install it but without success and it seems to be an ongoing problem that it is hard to get netcat running on Centos and Redhat distributions.

Error Message from salt '*' state.highstate:

ID: jenkins_listening
Function: pkg.installed
Name: netcat-openbsd
Result: False
Comment: The following package(s) were not found, and no possible matches were found in the package db: netcat-openbsd
Started: 09:21:59.906133
Duration: 14024.736 ms

Proposed solution:

  1. Either put a standardised way of installing netcat on Centos and Redhat distributions and everything else remain the same.
  2. Or to use another tool for checking the HTTP connection with jenkins server, maybe telnet or just curl

Current solution of mine:

I am using solution 2 and just use curl to test if connection with jenkins is there. I have to test if that works. I am extending your formula part for jenkins_listening.

Backward compatibility break: server_name is gone

Optionnal pillar jenkins.server_name was used to generate nginx configuration for jenkins.

Now, it's not functional anymore.

Apparently, BC was broke in 852f883, apparently because @nmadhok wanted to set a default pillar ? I'm probably mistaken. Why break BC and remove a feature when you just want to add a default ?

Variable timeout in cli.sls should be set by user using pillar

Sometimes users wish to restrict the timeout variable to wait for less seconds and sometimes for more. It would be better not to hard code this but rather to include it in the map.jinja so that users can use pillar to define it.

The default value should be still 360 if that is good based on experience. But 360 seconds seems to be a bit too long for me.

Provision jenkins with plugins

It would be great if jenkins-formula could install plugins. I have a poc a bit like this (edited for clarity) that could be worked out to be included in the formula:

{%- set cli = 'java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar' %}

jobs_dependencies:
  pkg.installed:
    - pkgs:
      - curl

jenkins_is_started:
  service.running:
    - name: jenkins

jenkins_port_bound:
  cmd.run: 
    - name: until curl -s -I http://localhost:8080 | grep X-Jenkins-CLI2-Port &> /dev/null; do sleep 1; done
    - stateful: True
    - require:
      - service: jenkins_is_started

jenkins_online:
  cmd.run:
    - name: while curl {{ jenkins.cli_url }} | grep -i restarting &> /dev/null; do sleep 1; done
    - stateful: True
    - env: 
      - JENKINS_URL: {{ jenkins.cli_url }}
    - require:
      - cmd: jenkins_port_bound

jenkins_updates_directory:
  file.directory:
    - name: /var/lib/jenkins/updates/
    - user: jenkins
    - group: jenkins
    - makedirs: True

jenkins_ready_for_plugins:
  cmd.run:
    - unless: test -f /var/lib/jenkins/updates/default.json
    - name: "wget http://updates.jenkins-ci.org/update-center.json -qO- | sed '1d;$d' > /var/lib/jenkins/updates/default.json"
    - require: 
      - cmd: jenkins_online
      - file: jenkins_updates_directory

{% for plugin in jenkins.plugins.installed %}
jenkins_plugin_{{ plugin }}:
  cmd.run:
    # we should rather call list-plugins once and then grep on the result in the loop
    - unless: {{ cli }} list-plugins | grep {{ plugin }}
    - name: {{ cli }} install-plugin {{ plugin }}
    - timeout: 360
    - env: 
      - JENKINS_URL: {{ jenkins.cli_url }}
    - require:
      - cmd: jenkins_ready_for_plugins 
{% endfor %}

jenkins_restart_for_plugins:
  service.running:
    - name: jenkins
    - watch:
      {% for plugin in jenkins.plugins.installed %}
      - cmd: jenkins_plugin_{{ plugin }}
      {% endfor %}

So as you can see there's a lot of boilerplate, and while this works it's far from perfect so we'd probably have to work on the PR for some weeks, but if we find this is within the scope of jenkins-formula then I could code that PR.

Plugins are not being installed

Salt is configured to run in stand-alone mode and has formula properly configured. I've created a pillar based on the example but added the plugins I need. When I run the following command it installs Jenkins but none of the plugins are installed:

sudo salt-call state.sls jenkins saltenv=ci

My pillar/jenkins.sls file is:

jenkins:
  lookup:
    port: 80
    home: /var/lib/jenkins
    user: jenkins
    group: www-data
    server_name: localhost
    master_url: http://localhost:8080
    plugins:
      installed:
        - any-buildstep
        - chroot
        - copyartifact
        - cron_column
        - cvs
        - environment-dashboard
        - environment-script
        - github-oauth
        - pollscm
        - publish-over-ssh
        - slack
        - ssh-slaves
        - subversion
        - thinbackup
        - translation
        - windows-slaves
    pkgs:
      - jenkins

Template ot Found on Debian Jessie

When I try and run this on my Debian Jessie server, I get this output:

vagrant@deploy:~$ sudo salt '*' state.highstate
[ERROR   ] Data passed to highstate outputter is not a valid highstate return: {'deploy.local': ['Rendering SLS \'base:apps.jenkins\' failed: Jinja error: jenkins/map.jinja\nTraceback (most recent call last):\n  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl\n    output = template.render(**decoded_context)\n  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render\n    return self.environment.handle_exception(exc_info, True)\n  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception\n    reraise(exc_type, exc_value, tb)\n  File "<template>", line 1, in top-level template code\n  File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 144, in get_source\n    raise TemplateNotFound(template)\nTemplateNotFound: jenkins/map.jinja\n\n; line 1\n\n---\n{% from "jenkins/map.jinja" import jenkins with context %}    <======================\n\njenkins_group:\n  group.present:\n    - name: {{ jenkins.group }}\n    - system: True\n[...]\n---']}
ERROR: Minions returned with non-zero exit code

installing jenkins.cli fails (for jenkins 2.16 on Ubuntu 14.04)

After running jenkins state and then running jenkins.cli, on state jenkins_responding the java call fails with a 403.

Reproducing it:
$ sudo java -jar /var/cache/jenkins/jenkins-cli.jar -s http://localhost:8080

java.io.IOException: No X-Jenkins-CLI2-Port among [X-Jenkins, null, Server, X-Content-Type-Options, X-You-Are-In-Group, X-Hudson, Date, X-Jenkins-Session, X-You-Are-Authenticated-As, X-Required-Permission, Set-Cookie, Expires, Content-Length, Content-Type]
at hudson.cli.CLI.getCliTcpPort(CLI.java:284)
at hudson.cli.CLI.(CLI.java:128)
at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72)
at hudson.cli.CLI._main(CLI.java:473)
at hudson.cli.CLI.main(CLI.java:384)
Suppressed: java.io.IOException: Server returned HTTP response code: 403 for URL: http://localhost:8080/cli

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.