lhupfeldt / jenkinsflow Goto Github PK
View Code? Open in Web Editor NEWPython API with high level build flow constructs for jenkins/hudson
License: BSD 3-Clause "New" or "Revised" License
Python API with high level build flow constructs for jenkins/hudson
License: BSD 3-Clause "New" or "Revised" License
It would be nice if jenkinsflow would be available in https://pypi.python.org
Hi, Lars
Just to let you know, I created quick and dirty implementation of new invocation type: invoke_delayed. In this invocation the job is not going to call jenkins for Job object till right before it will be invoked. This is to be able to call a job, that is being created by a job executed within pipeline.
Once I finish with it, I will create pull request.
Also, take a look at jenkinsapi, there is a new way to invoke jobs has been introduced recently.
Aleksey
Hi, Lars
I have RHEL 5.4:
lsb_release -i -r
Distributor ID: RedHatEnterpriseServer
Release: 5.4
when I try to python setup.py install
library subprocess32
fails to install due to C compilation issues. That library only used in tests and in one optional file that I am not going to use. So I have to modify setup.py
to remove that dependency...
Is it possible to remove that dependency?
Thanks!
Aleksey
I created simple test pipeline and execute two instances of it in parallel. When job is scheduled by one pipeline, but job status is QUEUED
second pipeline will "attach" to that build instead of scheduling another build. Once job has RUNNING
status - both pipelines will report the same build number.
I would expect that if build is queued the pipeline still try to schedule another build.
Thanks!
Aleksey
Hi,
It is not clear how to install this plugin into Hudson/Jenkins.
Is it even a plugin? If so - Hudson .hpi/Jenkins .jpi needs to be provided and plugin needs to be registered with Hudson/Jenkins plugin repos.
Hi,
Coming from use of jenkinsapi
, I expected that Jenkins.__init__
parameters would be in the same order as ones in jenkinsapi
. Apparently they are not. I think it would be a bit better to have the same order of params as in jenkinsapi
: url, username, password
and extra params would go next.
Aleksey
Lars,
I think it is bad to add this #!/usr/bin/python
into .py files. Some people don't have python in that location and some may have Python 2.4 in there.
Just a thought.
Hi!
In my Jenkins 1.484 I created 3 simple jobs: flow-1, flow-2 and flow-3. Each job does 'uname -a'.
I created simple flow:
with serial(api, timeout=70, job_name_prefix='ADMIN-', report_interval=3) as ctrl1:
ctrl1.invoke('flow-1')
with ctrl1.parallel(timeout=20, report_interval=3) as ctrl2:
ctrl2.invoke('flow-2')
ctrl2.invoke('flow-3')
ctrl1.invoke('flow-1')
And this happened:
Started by user 1459847
[EnvInject] - Loading node environment variables.
Building on master in workspace /shared/opt/jenkins-home/jobs/ADMIN-python-test/workspace
[workspace] $ /home/sabredev/virtualenv/jenkins/bin/python /tmp/shiningpanda1946216512484119647.py
['/tmp', '/home/sabredev/virtualenv/jenkins/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/sabredev/virtualenv/jenkins/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg', '/home/sabredev/virtualenv/jenkins/lib/python27.zip', '/home/sabredev/virtualenv/jenkins/lib/python2.7', '/home/sabredev/virtualenv/jenkins/lib/python2.7/plat-linux2', '/home/sabredev/virtualenv/jenkins/lib/python2.7/lib-tk', '/home/sabredev/virtualenv/jenkins/lib/python2.7/lib-old', '/home/sabredev/virtualenv/jenkins/lib/python2.7/lib-dynload', '/usr/local/scbpkg/lib/python2.7', '/usr/local/scbpkg/lib/python2.7/plat-linux2', '/usr/local/scbpkg/lib/python2.7/lib-tk', '/home/sabredev/virtualenv/jenkins/lib/python2.7/site-packages', '/home/sabredev/src']
== Legend ==
Serial builds: []
Parallel builds: ()
Invoking (w/x,y/z): w=current invocation in current flow scope, x=max in scope, y=total number of invocations, z=total max invocations
Elapsed time: 'after: x/y': x=time spent during current run of job, y=time elapsed since start of outermost flow
Queuing jobs for serial run: [
Queuing job: ADMIN-flow-1 'ADMIN-flow-1' Status IDLE - latest build: ADMIN-flow-1 #2
Queuing jobs for parallel run: (
Queuing job: ADMIN-flow-2 'ADMIN-flow-2' Status IDLE - latest build: ADMIN-flow-2 #1
Queuing job: ADMIN-flow-3 'ADMIN-flow-3' Status IDLE - latest build: ADMIN-flow-3 #1
)
Queuing job: ADMIN-flow-1 'ADMIN-flow-1' Status IDLE - latest build: ADMIN-flow-1 #2
]
Invoking (1/1,1/1): ['ADMIN-flow-1', ('ADMIN-flow-2', 'ADMIN-flow-3'), 'ADMIN-flow-1']
Invoking (1/1,1/1): http://sabrebuild1.uk.standardchartered.com:8080/job/ADMIN-flow-1 - /build
'ADMIN-flow-1' Status IDLE - latest build: ADMIN-flow-1 #2
'ADMIN-flow-1' Status IDLE - latest build: ADMIN-flow-1 #3
SUCCESS: 'ADMIN-flow-1' - build: http://sabrebuild1.uk.standardchartered.com:8080/job/ADMIN-flow-1/3/testReport/api/python/ after: 23.850s/23.850s
Invoking (1/1,1/1): ('ADMIN-flow-2', 'ADMIN-flow-3')
Invoking (1/1,1/1): http://sabrebuild1.uk.standardchartered.com:8080/job/ADMIN-flow-2 - /build
'ADMIN-flow-2' Status IDLE - latest build: ADMIN-flow-2 #2
SUCCESS: 'ADMIN-flow-2' - build: http://sabrebuild1.uk.standardchartered.com:8080/job/ADMIN-flow-2/2/testReport/api/python/ after: 17.901s/44.250s
Invoking (1/1,1/1): http://sabrebuild1.uk.standardchartered.com:8080/job/ADMIN-flow-3 - /build
'ADMIN-flow-3' Status IDLE - latest build: ADMIN-flow-3 #1
Traceback (most recent call last):
File "/tmp/shiningpanda1946216512484119647.py", line 41, in <module>
main()
File "/tmp/shiningpanda1946216512484119647.py", line 38, in main
ctrl1.invoke('flow-1')
File "/home/sabredev/src/jenkinsflow/jobcontrol.py", line 413, in __exit__
self.wait_for_jobs()
File "/home/sabredev/src/jenkinsflow/jobcontrol.py", line 383, in wait_for_jobs
last_report_time = self._check(start_time, last_report_time)
File "/home/sabredev/src/jenkinsflow/jobcontrol.py", line 331, in _check
last_report_time = job._check(start_time, last_report_time)
File "/home/sabredev/src/jenkinsflow/jobcontrol.py", line 263, in _check
self._check_timeout(start_time)
File "/home/sabredev/src/jenkinsflow/jobcontrol.py", line 229, in _check_timeout
raise FlowTimeoutException("Timeout after:" + self._time_msg(start_time) + unfinished_msg)
jenkinsflow.jobcontrol.FlowTimeoutException: Timeout after:after: 36.751s/63.099sUnfinished jobs:('ADMIN-flow-2', 'ADMIN-flow-3')
Build step 'Custom Python Builder' marked build as failure
[BFA] Scanning build for known causes...
[BFA] Done. 0s
Notifying upstream projects of job completion
Finished: FAILURE
Shall I increase timeout in this call?
with serial(api, timeout=70, job_name_prefix='ADMIN-', report_interval=3) as ctrl1:
Thanks!
Hi,
It doesn't work in latest Jenkins.
According to Jenkins Docs job cannot be triggered by issuing GET on $job_name/build URL. Instead, it now requires GET with token parameter. That parameter is set in additional setting in job itself and jenkinsapi will have to pass this token for the job. This is already supported by jenkinsapi, but here is no way to pass that token in jenkinsflow. Any parameters to invoke function will be treated as job parameters and in result jobs will fail to execute.
There shall be a way to pass token parameter as special one.
Hi, Lars
Somebody asked me to give details about job execution times in jenkinsflow pipeline. I think that it would be nice to add build execution time to a final report.
So it will look like this:
--- Final status ---
serial flow: [
serial flow: (
job: 'TDSS_Update' SUCCESS - 1m 30sec
job: 'TDSS_Update_adapters' SUCCESS - 30sec
job: 'TDSS_Update_eod' SUCCESS - 48sec
}
job: 'TDSS_Results' SUCCESS - 5 min 34 sec
]
To diagnose changes in complex flows, it would be helpfull to be able to dump a flow, wihtout executing it.
Output options could be pretty-print like in normal run output, and something that was easy to machine parse.
Hi, Lars
I have Jenkins served on "http://somehost/jenkins/", running under Apache. I've done upgrade to latest version and now nothing works because jenkinsflow asks Jenkins as if Jenkins in served from "http://somehost/".
Aleksey
Hi, Lars
One of the jobs in flow has failed, but the job, that runs flow itself shows that flow failed, but still returns success. How do I fail flow job when flow has failed?
--- Final status ---
serial flow: [
serial flow: [
job: 'JOB_1' SUCCESS
job: 'LONG_JOB' FAILURE http://jenkins/jenkins/job/LONG_JOB/208/console
job: 'JOB_2' UNKNOWN - IDLE
]
parallel flow: (
serial flow: [
job: 'JOB_3' UNKNOWN - IDLE
job: 'JOB_4' UNKNOWN - IDLE
]
serial flow: [
job: 'JOB_5' UNKNOWN - IDLE
job: 'JOB_6' UNKNOWN - IDLE
]
)
]
Finished: SUCCESS
Note that flow failed, but jenkinsflow
still returns "SUCCESS" instead of "FAILURE".
Is there something I miss?
Hi, Lars
I tried to run demos and found that they require proxytools
package, which is listed as test
dependency only. I think that demos shall run either w/o that package or that package have to be listed as normal dependency.
Aleksey
Hi, Lars
Another enchancement idea. Imagine that I want to get build results and timings in some report. For that I would like to call some method of jenkinsflow, to return current flow state.
It is impossible to do so now because any of the methods that may have that information are simply printing it.
Another possibility of doing that is to register fake job invocation...
Aleksey
It would be nice to have documentation to be available on ReadTheDocs.com or on PythonHosted.com - AFAIK they both support Sphinx generated docs and both are free.
Hi, Lars
I have big pipeline, and I want to start running it even if some jobs from previous execution are still running.
I know that once execution will come to those jobs, they will be finished already and nothing will be out of sync.
Is there a parameter to do that?
Thanks!
Hi, Lars,
Another suggestion: I have lenghty subflow, and I want to send email once that flow have finished. If I add email function inside with serial(...)
- it will be called right when flow is parsed. So I have to make send_email job and call it from flow. Kind of inneficient.
What I suggest is to have an ability to register callback functions with the flow:
with serial(...) as main_flow:
main_flow.on_success(some_callable)
main_flow.on_failure(some_other_callable)
main_flow.invoke('job1')
That way I don't have to do a separate job and can do interesting things from the same script.
Thanks!
Aleksey
When a flow fails, it can be hard to diagnose which jobs did not get run and if multiple jobs failed.
To make this easier - I propose that on failure the job flow (which is pretty-printed at the start of the run), is re printed, in pretty format, but including job status:
serial flow: [
parallel flow: (
job: a_job_name: SUCCESS
job: b_job_name: FAILURE
)
job: c_job_name: NOT_RUN
]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.