Development flow about grid

[coppolachan]

Looks cool, have you already tried it?

G

On 16/01/16 01:09, Antonin Portelli wrote:

Hi,

Recently my attention was attracted to the following extension of git
to manage the development flow of a project (features, stable/unstable
branches, ): http://danielkummergithubio/git-flow-cheatsheet/

I thought it was worth considering it regarding the recent discussion
and Guido's suggestion (which I completely support) of some form
quality control and maybe milestones

Let me know what you think

—
Reply to this email directly or view it on GitHub
#23.

######################################################
Guido Cossu
KEK Theory Center
Institute of Particle and Nuclear Studies
High Energy Accelerator Research Organization KEK
1-1 Oho, Tsukuba-shi,Ibaraki,305-0801 JAPAN
Room: 3rd Floor n. 304
Phone: [phone]
email: cossu(AT)post(DOT)kek(DOT)jp
######################################################

[paboyle]

does indeed look cool -- I'll think about trying a little prototype (not grid) and see
how I like it. Any experience?

[paboyle]

Will add to this issue, rather than file a new one.

I've set up a Cron job on Cori to run nightly builds

icpc-none-avx2
icpc-mpi-avx2
g++-none-avx2
g++-mpi-avx2
icpc-none-avx512

questions:

• complete list of targets? Will add clang at some point, once they have clang 3.8
  guess I could add avx, sse, but worried about 3 compilers x 4 intrinsics frameworks x 2 comms = 24
• performance checks worth adding I guess?
• correctness checks worth adding
• CC email on fail
• CC email on success
• Report back to github HQ ? How?

[paboyle]

and p.s. annoyed at github -- they should provide continuous integration support; just lazy to
acquire all this open source and invest the minimum. They don't deserve our custom...

[mspraggs]

There are other free providers, e.g. https://travis-ci.org/. EDIT: And it has GitHub hooks so that it only runs a new test when new commits are pushed.

[aportelli]

Please find below some random thoughts about a development workflow:

• No I do not have experience with git-flow, to use it we probably need to figure out a couple of not-so-trivial things (version numbering, definition of 'stable', ...). I agree that playing with a prototype is a good idea, I can also do that and report back here.
• For continued integration I do not have a strong opinion about hosting. Just one little thing about Cori: I guess it is tricky to have access to it and that the lifetime of this machine will be finite of the order of some years. Maybe we want to use a machine we have more control on? I can think about our Xeon server (in Soton, with icpc). The service Matt pointed out looks also interesting, I am just wondering: what are the fine prints associated with the fact that this is a free service?
• For CI workflow: I think 24 different built is alright, it can help us checking that we are not breaking any old code by adding new things. I totally support the idea of performance & consistency checks, maybe going for a proper unit test setup is a bit much but something that approximates it can be good.
• Related to CI workflow: I am already using with some Soton students (including Matt) Slack to communicate about code and technical issue. One could easily complain about the fact that it is yet another communication channel, but in my experience it is really excellent to keep track of all technical issues going on. Slack offers endless automated integration with external services. For example when I update my analysis code the commit summary is automatically posted from GitHub/BitBucket to a channel where Matt & I are talking about it. When a rare kaon job on the BG/Q starts or finishes, a message is automatically posted to a channel where Andrew & I are working. Slack could be useful for Grid development, and can be a place where the CI reports are sent. I am happy to work on setting up something like this if people think it is a good idea.

[mspraggs]

To answer your question RE Travis, I don't think there's any limitations for open source projects beyond "fair use" (see https://travis-ci.com/plans). Obviously if you're sucking up horrendous numbers of core hours for many different build configurations, then that may cause problems. I use it for pyQCD, though admittedly the number of files to compile is much smaller.

EDIT: More info on how the Travis CI queue works here.

[aportelli]

Hi,

Just an update about the source code development flow. I have tried git-flow and I think it is really good. It is essentially just a compilation of "high-level" git scripts that implement the development flow strategy described here http://nvie.com/posts/a-successful-git-branching-model/. This is an interesting article to read. I had a quick chat with Peter today and it sounds it could be a useful way to organise the development.

Maybe a good way to discuss that would be to have a meeting once Guido is here in Edinburgh?

[paboyle]

I have a travis configured on g++-4.9 and AVX2.

We can grow this to clang-3.8++, and also various SSE4, AVX, AVX2 I guess.

We still need ICPC compiles, and the nightly at NERSC will remain in addition to Travis-CI hooks.

[aportelli]

I guess this issue is solved?

[coppolachan]

I will add that the builds should be 2 times 24. We need a build for the single precision and one for the double in order to spot locations where an assumption on the precision is made that clashes with the general definition. I found one of these yesterday for example.

[paboyle]

I added double and single to the build and test.
We have both CI and git flow in use now.
Closing.