biddisco / pycicle Goto Github PK
View Code? Open in Web Editor NEWPython Continuous Integration Command Line Engine
Python Continuous Integration Command Line Engine
When builds are started, currently, we use a fixed set of options for each 'machine'. A new config file with a different machine name can be used to create a second set of options for a the same machine, but it would be better to allow many options/choices to be setup and allow pycicle to pick some either at random - or according to some other rule.
Options would generally fall into categories
WITH_XXX
)Recent builds on CSCS machine daint failed with an error
sbatch: error: You are unable to submit jobs because you have exceeded the quota of
1 million off files/folders (inodes) on the scratch file system.
You can find more details about the quotas on the CSCS User Portal ...
There should be a cleanup step added that will either
The second option is currently preferred for simplicity.
When adding new features to pycicle, build/test fails that are caused by incorrect setup can make a project look bad by setting failed status on gihub PRs. There should be a --no-status
or -n
flag to disable status setting during testing of pycicle itself.
I think from readying this that for a regex like in our code that using
r"whatever" is the correct replacement for ur I think.
Using python 2.7.x
I find that only PYCICLE_GITHUB_TOKEN is all that works. I suspect this might be a unicode issue. Did some fooling with that, inconclusive.
Going to try pycicle in python 3, if that just works perhaps pycicle should just go python 3 only.
There is a bad smell around PYCICLE_GITHUB_BASE_BRANCH & PYCICLE_BASE
Just seems like a potentially disasterous alias for PICYCLE_BASE or vice versa
as far as I can tell they are set equal here and that's that. Could be for the others as well. Seems like the intention was to decouple the PYCICLE_GITHUB values from those passed to the CDash dashboard. But then GITHUB_ORG and GITHUB_PROJ are reused.
What about treating all variables/defines coming in from cmake configs as a dictionary with keys that are the cmake variable names. Most could just be carried straight across to the dashboard call.
In order to use pycicle for projects other than HPX, there are some HPX specific variables/links that need to be made into options or configurable elements.
Currently pycicle does not wipe binary dirs between builds. this can lead to invalid CMakeCache states that invalidate build/test results.
Since hpx is the build canary
Hello,
Your project was referred to me by Hannes Vogt, and I am considering using pycicle
in our organization (Polish meteorological services, IMGW) for testing development of our new dynamic core for the weather forecast.
Our project is built and tested exclusively by CMake and CTest (with make
and make test
). Besides C++ and CUDA it does have Fortran code. The git repository containing the project itself contains git submodules. The project itself is hosted on our internal GitLab instance, and some submodules are on GitHub. We wish to build the project on our supercomputer managed by the Torque (not Slurm), where compute nodes do not have access to the internet. We don't use CDash (but may start using it, if it is recommended).
Do you think this kind of project is within reach of the pycicle
(or at least pycicle
can be easily expanded to cover it)?
This will include refactoring master => base MASTER => BASE where appropriate to avoid confusion. It should not break hpx.
When pycicle checks out a new branch to test a PR, it will not get the right SHA when the PR comes from a remote clone of the repo being tested.
The sequence used here assumes that the branch being tested comes from origin and therfore the SHA will not be fetched for PR from a clone
${CTEST_GIT_COMMAND} checkout ${PYCICLE_MASTER};
${CTEST_GIT_COMMAND} fetch origin;
${CTEST_GIT_COMMAND} reset --hard origin/${PYCICLE_MASTER};
${CTEST_GIT_COMMAND} branch -D ${GIT_BRANCH};
${CTEST_GIT_COMMAND} checkout -b ${GIT_BRANCH};
${CTEST_GIT_COMMAND} fetch origin ${PYCICLE_BRANCH};
${CTEST_GIT_COMMAND} merge --no-edit FETCH_HEAD;
${CTEST_GIT_COMMAND} checkout ${PYCICLE_MASTER};
${CTEST_GIT_COMMAND} clean -fd;"
Allow pycicle to take its per project settings from the source project itself.
Each project that uses pycicle for testing can have a pycicle subdir that stores the testing/config settings, the main advantage being
and lots more reasons
When the test project is first configured, ctest submits results and pycicle is updating not just the config, but also build and test status with 'ok' before they have actually run. They are correctly set later to 'fail' if there are problems, but they should be marked as running rather than ok.
Currently, the user's home directory gets all the slurm-123456.out
and slurm-123456.err
files. It ought to be able to put them somewhere else. A better default would be $PYCICLE_ROOT/temp
with the option to put them somewhere else as a command-line or config setting.
It would be useful to be able to pass a set of build options to the configs to easier run builds with combinations of options. These would be for example:
These can be generic though so that projects can define their own set of options. I'm not sure what the best syntax would be for this (could be command line options or a separate build options config file), but essentially one should be able to launch pycicle with:
project=hpx
machine=daint
build_options={compiler: gcc49, clang5; generic_option_1: value_1, value_2; generic_option_2: value_1, value_2}
pycicle would then generate the cartesian product of the build options and launch a build for each combination with ctest -S config/hpx/daint.cmake ... -Dcompiler=gcc49 -Dgeneric_option_1=value_1 -Dgeneric_option_2=value_1
.
The build name should be derived from the build options to make them unique.
Conflicts/dependencies between build options are project specific anyway, so this should be handled on a per project basis in the config files.
One may not want to run builds for all combinations of build options so one could additionally:
Ideally the build options and config (daint.cmake
) would be taken from the source repository but this might require a bigger refactoring.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.