saltstack-formulas / jenkins-formula Goto Github PK
View Code? Open in Web Editor NEWHome Page: http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html
License: Other
Home Page: http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html
License: Other
The jenkins.conf file is configured for RedHat and does not work on Debian based systems.
Here is a pull request to resolve the issue.
Hi,
Here at PeopleDoc we built a salt formula for deploying master/slave jenkins. Our target is to fully provision jenkins through salt. We open sourced the code at https://github.com/novafloss/ci-formula . What do you thing of merging ci-formula in jenkins-formula to address more use case.
In the end we would have jenkins.master
and jenkins.slave
entry point. We could also add jenkins_job
, jenkins_view
and jenkins_plugin
states, but this require jenkins-cli
.
If you are ok, i can do some pull requests to progressively merge interesting feature in official jenkins-formula.
Regards,
Γtienne
Currently the formula does not work for cent7, because java needs to be installed.
See #27 for some solutions.
Just a slight suggestion and I also wish to ask for your opinion about this:
There are a number of macros in the sls files. How about we put them into a single file called macros and comment each macros what it does. Then import them with a single line into any states files if they are needed.
A better approach may yet be to separate the macro files based on topic:
macro-cli
macro-jenkins
macro-etc...
This way the user knows exactly which macro is written in which file and if someone just need to update one, they can do so without messing around with the other files.
This will, however, only add more benefit if this project becomes larger and we have more macros to maintain. To make it future proof, I do suggest we start now with this.
----------
ID: jenkins
Function: pkg.installed
Result: False
Comment: Error occurred installing package(s). Additional info follows:
errors:
- Running scope as unit: run-ra5684b079467415dbb0c6ac1ff786386.scope
Last metadata expiration check: 0:00:02 ago on Mon Mar 25 19:48:31 2024.
Dependencies resolved.
================================================================================
Package Architecture Version Repository Size
================================================================================
Installing:
jenkins noarch 2.440.2-1.1 jenkins 82 M
Transaction Summary
================================================================================
Install 1 Package
Total download size: 82 M
Installed size: 83 M
Downloading Packages:
jenkins-2.440.2-1.1.noarch.rpm 45 MB/s | 82 MB 00:01
--------------------------------------------------------------------------------
Total 44 MB/s | 82 MB 00:01
Jenkins upstream package repository 40 kB/s | 3.1 kB 00:00
Importing GPG key 0x45F2C3D5:
Userid : "Jenkins Project <[email protected]>"
Fingerprint: 62A9 756B FD78 0C37 7CF2 4BA8 FCEF 32E7 45F2 C3D5
From : https://pkg.jenkins.io/redhat-stable/jenkins.io.key
Key imported successfully
Import of key(s) didn't help, wrong key(s)?
Public key for jenkins-2.440.2-1.1.noarch.rpm is not installed. Failing package is: jenkins-2.440.2-1.1.noarch
GPG Keys are configured as: https://pkg.jenkins.io/redhat-stable/jenkins.io.key
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED
Started: 19:48:26.886320
Duration: 9909.231 ms
Changes:
----------
gpg-pubkey.(none):
----------
new:
3228467c-613798eb,350d275d-6279464b,45f2c3d5-5e81efb9,621e9f35-58adea78,73d76b3f-63dace60,7bd9bf62-5762b5f8
old:
3228467c-613798eb,350d275d-6279464b,621e9f35-58adea78,73d76b3f-63dace60,7bd9bf62-5762b5f8
- jenkins-formula:
rev: 'da763dd' # 2024.03 - git rev-parse --short HEAD, tags are not available for the repo
Problem line is https://github.com/saltstack-formulas/jenkins-formula/blob/da763dd88f0b7871ff01272b3a894e0bebf680f7/jenkins/init.sls#L46C7-L46C76
- gpgkey: https://pkg.jenkins.io/redhat{{ repo_suffix }}/jenkins.io.key
should be replaced with line
- gpgkey: https://pkg.jenkins.io/redhat{{ repo_suffix }}/jenkins.io-2023.key
or move key name to defaults so it could be changed via lookup
I have tried to install plugins with default config.xml file on debian 8, but jenkins cli doesn't work due to error: No X-Jenkins-CLI2-Port among.
I have fix it in config.xml - changed value from default <slaveAgentPort>-1</slaveAgentPort>
to 0 (random port) a restart jenkins service before jenkins_listening
Tested on jenkins version 2.7/2.8 instaled by this formula.
Does someone have same problem with plugins/cli or some better solution?
Just curious if it makes any difference to put the jinja imports before the YAML include found in jobs.sls for example.
What is the best practice if if it does not make a difference, which one should we stick to?
Add a setting to choose the LTS release version instead of the regular release
Having these hardcoded:
- user: www-data
- group: www-data
prevents cross-distro use. I'll open a PR.
Hi,
When running this formula on a very clean box, such that it minimal install doesn't include a Java version. The formula breaks in an somewhat goofy way: Jenkins has dependency on Java to run. Perhaps include a test that Java is installed and at some reasonable minimal version ?
or a doc change that suggests including: https://github.com/saltstack-formulas/java-formula
There should be a way of providing jenkins credentials in order to use jenkins-cli. Jenkins-cli uses rest calls to install plugins. jenkins.plugins
state will basically break as soon as you setup security on jenkins.
e.g. I would like to use jenkins github oauth plugin for authentication: https://wiki.jenkins-ci.org/display/JENKINS/Github+OAuth+Plugin
There should be a way of passing github oauth token for connections between jenkins-cli and local jenkins instance. All credentials should be provided in pillar.
With newer versions of Jenkins, now have setup wizard to contend with. Found a decent way to override that by adding the following to map.jinja (java args line 11):
'java_args': '-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false',
seems there are other groovy scripts to disable the anonymous user and turn on security that we could somehow include ?
see:
https://groups.google.com/forum/#!msg/jenkinsci-users/Pb4QZVc2-f0/ywKqZVf9MgAJ
Here is a simple hack to install plugins, maybe someone likes it.
It is working now, needs some settings to be installed.
you will also need to restore /var/lib/jenkins/jenkins_settings/installedPlugins.xml and then
you will have a bunch of things like /var/lib/jenkins/*.xml that need to be setup.
{% set plugins = ("analysis-collector","analysis-core","ansicolor","ant","artifactdeployer","artifactory","batch-task","build-metrics","build-publisher","checkstyle","chucknorris","cobertura","conditional-buildstep","configurationslicing","confluence-publisher","copyartifact","credentials","cvs","dashboard-view","deploy","disk-usage","email-ext","envinject","extended-choice-parameter","external-monitor-job","extra-columns","fail-the-build-plugin","git-client","git-server","git","global-build-stats","greenballs","groovy-postbuild","groovy","htmlpublisher","javadoc","jenkinswalldisplay","jira","jobConfigHistory","jquery","ldap","log-parser","mailer","mask-passwords","maven-info","maven-plugin","multiple-scms","nested-view","nodelabelparameter","pam-auth","parameterized-trigger","pmd","promoted-builds-simple","promoted-builds","publish-over-ssh","python","run-condition","scm-api","scm-sync-configuration","scp","scriptler","sectioned-view","selenium-aes","selenium","seleniumhq","shelve-project-plugin","sidebar-link","sounds","ssh-agent","ssh-credentials","ssh-slaves","ssh","subversion","svn-tag","svnpublisher","testng-plugin","text-finder","thinBackup","token-macro","translation","view-job-filters","ws-cleanup") %}
{% for plugin in plugins %}
jenkins_install_plugin_{{plugin}}_1:
cmd.run:
- name : curl -k -L https://updates.jenkins-ci.org/latest/{{plugin}}.hpi -o /var/lib/jenkins/plugins/{{plugin}}.hpi
- onlyif: 'test ! -f /var/lib/jenkins/plugins/{{plugin}}.hpi'
jenkins_install_plugin_{{plugin}}_2:
file.managed:
- {name: "/var/lib/jenkins/plugins/{{plugin}}.hpi"}
- {user: 'jenkins'}
- {group: 'jenkins'}
- {replace : False }
- require:
- cmd : jenkins_install_plugin_{{plugin}}_1
jenkins_install_plugin_{{plugin}}_3:
archive.extracted:
- name: "/var/lib/jenkins/plugins/{{plugin}}/"
- source: "/var/lib/jenkins/plugins/{{plugin}}.hpi"
- archive_format: zip
- user: jenkins
- group: jenkins
- onlyif: 'test ! -f /var/lib/jenkins/plugins/{{plugin}}'
- require:
- file : jenkins_install_plugin_{{plugin}}_2
{% endfor %}
kitchen converge is not working. First thing was it wanted me to create the vendor_path, mkdir vendor
No it can't find dependencies and is failing, I'm fairly new to salt but have used Kitchen (with Chef) a bunch. Any advice about how to best tackle this. I know kitchen-salt has dependencies but I can't get them to work:
Currently, the server_name nginx configuration variable for jenkins is commented.
It would be great if we could override it, so that other sites can live along with jenkins on this nginx configuration.
Problem:
When I try to use jenkins.nginx to install nginx config for jenkins, it does not work on Centos 7 since sites-available and enabled are not the correct folders.
Solution:
I used this custom state instead:
# Install nginx, run nginx only if installation is successful
nginx:
pkg.installed: []
/etc/nginx/default.d/jenkins.conf:
file.managed:
- source: salt://nginx/files/jenkins.conf
- user: nginx
- group: nginx
- require:
- pkg: nginx
/etc/nginx/nginx.conf:
file.managed:
- source: salt://nginx/files/nginx.conf
- user: nginx
- group: nginx
- require:
- pkg: nginx
Run nginx when everything is ready:
service.running:
- name: nginx
- enable: True
- reload: True
- require:
- pkg: nginx
- watch:
- file: /etc/nginx/default.d/jenkins.conf
- file: /etc/nginx/nginx.conf
The files are placed at the usual directories within the nginx directory I created.
Proposed Solution:
To get rid of the hard coded version and make further centos and redhat friendly version available through map.jinja and pillar. Make the source file pillar configurable and the map.jinja include variables for the final path of clients depending on the linux distribution.
This for now is mainly to resolve the netcat-related issues that have come up around #47 - as netcat (the package) is named differently and (the binary) has different flags it looks like a good solution to me to add platform-specific settings to map.jinja - i'll follow up with a PR asap.
remove nginx within the jenkins formula, move to include and pillars
Problem:
When using jenkins to install Jenkins I always get the development version but not the stable LTS. I wish to choose which one to install.
Solution:
I used the following to extend your formula:
extend:
jenkins:
{% if grains['os_family'] in ['RedHat', 'Debian'] %}
pkgrepo.managed:
- humanname: Jenkins upstream package repository
{% if grains['os_family'] == 'RedHat' %}
# Upgrade to Jenkins 2.7 LTS
- baseurl: http://pkg.jenkins-ci.org/redhat-stable
- gpgkey: http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
{% elif grains['os_family'] == 'Debian' %}
- file: {{jenkins.deb_apt_source}}
- name: deb http://pkg.jenkins-ci.org/debian binary/
- key_url: http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key
{% endif %}
- require_in:
- pkg: jenkins
{% endif %}
pkg.installed:
- pkgs: {{ jenkins.pkgs|json }}
service.running:
- enable: True
- watch:
- pkg: jenkins
Proposed Solution:
I wish to propose that you add a variable for the repo distro, i think it is just an one liner. Include this in the map.jinja and pillar so users can use pillar to choose whether to install LTS or the dev version of Jenkins.
If I have the time, I will definitely make a PR. Let me know if you have the time to do it yourself soon, otherwise, I can try to make some time on my side asap. Just wish to know your time schedule so I don't work on the same thing as you do.
We recently started using jenkins-formula for managing all aspects of Jenkins.
When an XML job def is edited, the new file is copied to the server, but the job is not updated and therefore the new configuration is not used.
Steps to reproduce:
I have an incoming PR to fix this.
First off, many thanks for putting the effort to create this formula, I am using it at my work and it is a bliss.
The problem:
When I used jenkins.plugins to install the plugins for Jenkins, jenkins_listening returns false since netcat-openbsd is not installed on Centos 7. I tried to install it but without success and it seems to be an ongoing problem that it is hard to get netcat running on Centos and Redhat distributions.
Error Message from salt '*' state.highstate:
ID: jenkins_listening
Function: pkg.installed
Name: netcat-openbsd
Result: False
Comment: The following package(s) were not found, and no possible matches were found in the package db: netcat-openbsd
Started: 09:21:59.906133
Duration: 14024.736 ms
Proposed solution:
Current solution of mine:
I am using solution 2 and just use curl to test if connection with jenkins is there. I have to test if that works. I am extending your formula part for jenkins_listening.
Optionnal pillar jenkins.server_name
was used to generate nginx configuration for jenkins.
Now, it's not functional anymore.
Apparently, BC was broke in 852f883, apparently because @nmadhok wanted to set a default pillar ? I'm probably mistaken. Why break BC and remove a feature when you just want to add a default ?
Sometimes users wish to restrict the timeout variable to wait for less seconds and sometimes for more. It would be better not to hard code this but rather to include it in the map.jinja so that users can use pillar to define it.
The default value should be still 360 if that is good based on experience. But 360 seconds seems to be a bit too long for me.
It would be great if jenkins-formula could install plugins. I have a poc a bit like this (edited for clarity) that could be worked out to be included in the formula:
{%- set cli = 'java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar' %}
jobs_dependencies:
pkg.installed:
- pkgs:
- curl
jenkins_is_started:
service.running:
- name: jenkins
jenkins_port_bound:
cmd.run:
- name: until curl -s -I http://localhost:8080 | grep X-Jenkins-CLI2-Port &> /dev/null; do sleep 1; done
- stateful: True
- require:
- service: jenkins_is_started
jenkins_online:
cmd.run:
- name: while curl {{ jenkins.cli_url }} | grep -i restarting &> /dev/null; do sleep 1; done
- stateful: True
- env:
- JENKINS_URL: {{ jenkins.cli_url }}
- require:
- cmd: jenkins_port_bound
jenkins_updates_directory:
file.directory:
- name: /var/lib/jenkins/updates/
- user: jenkins
- group: jenkins
- makedirs: True
jenkins_ready_for_plugins:
cmd.run:
- unless: test -f /var/lib/jenkins/updates/default.json
- name: "wget http://updates.jenkins-ci.org/update-center.json -qO- | sed '1d;$d' > /var/lib/jenkins/updates/default.json"
- require:
- cmd: jenkins_online
- file: jenkins_updates_directory
{% for plugin in jenkins.plugins.installed %}
jenkins_plugin_{{ plugin }}:
cmd.run:
# we should rather call list-plugins once and then grep on the result in the loop
- unless: {{ cli }} list-plugins | grep {{ plugin }}
- name: {{ cli }} install-plugin {{ plugin }}
- timeout: 360
- env:
- JENKINS_URL: {{ jenkins.cli_url }}
- require:
- cmd: jenkins_ready_for_plugins
{% endfor %}
jenkins_restart_for_plugins:
service.running:
- name: jenkins
- watch:
{% for plugin in jenkins.plugins.installed %}
- cmd: jenkins_plugin_{{ plugin }}
{% endfor %}
So as you can see there's a lot of boilerplate, and while this works it's far from perfect so we'd probably have to work on the PR for some weeks, but if we find this is within the scope of jenkins-formula then I could code that PR.
Salt is configured to run in stand-alone mode and has formula properly configured. I've created a pillar based on the example but added the plugins I need. When I run the following command it installs Jenkins but none of the plugins are installed:
sudo salt-call state.sls jenkins saltenv=ci
My pillar/jenkins.sls file is:
jenkins:
lookup:
port: 80
home: /var/lib/jenkins
user: jenkins
group: www-data
server_name: localhost
master_url: http://localhost:8080
plugins:
installed:
- any-buildstep
- chroot
- copyartifact
- cron_column
- cvs
- environment-dashboard
- environment-script
- github-oauth
- pollscm
- publish-over-ssh
- slack
- ssh-slaves
- subversion
- thinbackup
- translation
- windows-slaves
pkgs:
- jenkins
The pkgrepo.managed line breaks this formula on systems that don't support pkgrepo.managed. The state directive for the jenkins upstream pkg repo should probably be in its own file and included based on a configuration knob.
When I try and run this on my Debian Jessie server, I get this output:
vagrant@deploy:~$ sudo salt '*' state.highstate
[ERROR ] Data passed to highstate outputter is not a valid highstate return: {'deploy.local': ['Rendering SLS \'base:apps.jenkins\' failed: Jinja error: jenkins/map.jinja\nTraceback (most recent call last):\n File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 368, in render_jinja_tmpl\n output = template.render(**decoded_context)\n File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render\n return self.environment.handle_exception(exc_info, True)\n File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception\n reraise(exc_type, exc_value, tb)\n File "<template>", line 1, in top-level template code\n File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 144, in get_source\n raise TemplateNotFound(template)\nTemplateNotFound: jenkins/map.jinja\n\n; line 1\n\n---\n{% from "jenkins/map.jinja" import jenkins with context %} <======================\n\njenkins_group:\n group.present:\n - name: {{ jenkins.group }}\n - system: True\n[...]\n---']}
ERROR: Minions returned with non-zero exit code
After running jenkins state and then running jenkins.cli, on state jenkins_responding the java call fails with a 403.
Reproducing it:
$ sudo java -jar /var/cache/jenkins/jenkins-cli.jar -s http://localhost:8080
java.io.IOException: No X-Jenkins-CLI2-Port among [X-Jenkins, null, Server, X-Content-Type-Options, X-You-Are-In-Group, X-Hudson, Date, X-Jenkins-Session, X-You-Are-Authenticated-As, X-Required-Permission, Set-Cookie, Expires, Content-Length, Content-Type]
at hudson.cli.CLI.getCliTcpPort(CLI.java:284)
at hudson.cli.CLI.(CLI.java:128)
at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:72)
at hudson.cli.CLI._main(CLI.java:473)
at hudson.cli.CLI.main(CLI.java:384)
Suppressed: java.io.IOException: Server returned HTTP response code: 403 for URL: http://localhost:8080/cli
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.