Expand travis tests to multiple Linux distros about mb-system

[berkowski]

Ubuntu Focal doesn't seem too bad and I think I've gotten most of the linux build iterations working now. I've put together a few tables of what's being built at

https://github.com/berkowski/MB-System/blob/master/ci/README.md

[berkowski]

@schwehr, any thoughts? Builds are ticking over at https://travis-ci.com/berkowski/MB-System . Docker images are at https://hub.docker.com/r/zberkowitz/mbsystem-deps/tags . The framework for the single-repo approach seems to be working, so figuring out what you'd like for coverage and when things are built would seem to be the next step.

[dwcaress]

Zac, In the four builds you are doing, what are the ranges of versions of Proj and GMT that are involved? At present, we need the code to work with Proj 4.8 or 4.9, Proj 5.2, and Proj 6.1 or later. We also need the code to work with GMT 5.4.5 and GMT 6.0. Because there are API changes involved, there are corresponding preprocessor directives in the MB-System source that need testing. Dave

…

On Jan 17, 2020, at 7:32 AM, Zac Berkowitz ***@***.***> wrote: @schwehr, any thoughts? Builds are ticking over at https://travis-ci.com/berkowski/MB-System . Docker images are at https://hub.docker.com/r/zberkowitz/mbsystem-deps/tags . The framework for the single-repo approach seems to be working, so figuring out what you'd like for coverage and when things are built would seem to be the next step. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[berkowski]

Dave,

Right now I'm just using whatever is available as "standard" for each repo, except for Xenial which doesn't have a new enough version of GMT:

• Ubuntu 19.04
  • GCC: 8.3.0
  • PROJ: 5.2.0
  • GMT: 5.4.5
• Ubuntu 18.04
  • GCC: 7.4.0
  • PROJ: 4.9.3
  • GMT: 5.4.3
• Ubuntu 16.04
  • GCC: 5.4.0
  • PROJ: 4.9.2
  • GMT: 6.0.0 (from source)
• CentOS 7
  • GCC: 4.8.5
  • PROJ: 4.8.0
  • GMT: 5.4.4

I think if you want to test against specific combinations of PROJ and GMT it'd be best to pick a single distribution and install those project versions from source. That's easily doable.

[dwcaress]

Zac, I don’t think we have to do all possible permutations, but a couple of additional variants will be useful. Specifically, we should do an Ubuntu 19 and a CentOs 7 that always use the most current Proj (now 6.3) and GMT (now 6.0). Once GDAL and GMT are available for CentOs 8, we should make the most current CentOs case be 8 instead of 7. There will, naturally, be another Proj release within the next couple of weeks. I don’t think the parts of the API we use are changing this time, but I’ve been repeatedly surprised. Thanks, Dave

…

On Jan 17, 2020, at 8:44 AM, Zac Berkowitz ***@***.***> wrote: Dave, Right now I'm just using whatever is available as "standard" for each repo, except for Xenial which doesn't have a new enough version of GMT: • Ubuntu 19.04 • GCC: 8.3.0 • PROJ: 5.2.0 • GMT: 5.4.5 • Ubuntu 18.04 • GCC: 7.4.0 • PROJ: 4.9.3 • GMT: 5.4.3 • Ubuntu 16.04 • GCC: 5.4.0 • PROJ: 4.9.2 • GMT: 6.0.0 (from source) • CentOS 7 • GCC: 4.8.5 • PROJ: 4.8.0 • GMT: 5.4.4 I think if you want to test against specific combinations of PROJ and GMT it'd be best to pick a single distribution and install those project versions from source. That's easily doable. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[berkowski]

Sure. I'll start putting together two more images with the intention on tracking the latest versions of PROJ and GMT on Ubuntu and CentOS and update the list.

[schwehr]

Anything with a 2nd config is better that the single configuration that we have now, so wishing... :)

My wish for the short term would be to have builds that cover at least once the following. The GEOS CI system builds 33 different configs, but that's nuts.

• MB-System configs
  • With gsf, with trn*, default debugging and optimization (-g -O2)
  • disable-gsf, without trn, -g0 no debugging, -O3 aggressive optimization, -DNDEBUG
• Compiler
  • recent llvm/clang
  • recent gcc
• OS
  • ubuntu 20.04LTS
  • ubuntu 18.04LTS
  • CentOS something
  • Mac OSX whatever is easiest on Travis
• PROJ
  • 6.3
  • 5.[2-9].[0-9]
  • 4.9.?
  • Eventually Proj 7
• GMT
  • 6.x
  • 5.4.x

I think we should not have a Ubuntu 16.04LTS build. And as soon as there is an Ubuntu 20.04LTS, we should not have 19.04 or 19.10.

[berkowski]

I've got an image with Ubuntu 18.04 using PROJ 6.3, GMT 6.0.0, and GDAL 3.0.3 now. CentOS 7 would require building sqlite3 from sources to satisfy PROJ 6.3 requirements. The machinery is there for CentOS 8, but I guess I'd like to see what their pre-built packages are going to look like first.

For selfish reasons I'd like to keep Ubuntu 16.04 on the list, though I don't think it needs to be built every push to master. I'd be quite happy with it being built only for new tag releases as part of the PR approval. Could even make it "fail ok"

I think the individual MB-System config builds (-O2, -O3, etc) should use a single base image, along with the clang target and the build case against the latest PROJ and GMT. Do you have a preference?

Anything with a 2nd config is better that the single configuration that we have now, so wishing... :)

My wish for the short term would be to have builds that cover at least once the following. The GEOS CI system builds 33 different configs, but that's nuts.

* MB-System configs
  
  * With gsf, with trn*, default debugging and optimization (-g -O2)
  * disable-gsf, without trn, -g0 no debugging, -O3 aggressive optimization, -DNDEBUG

* Compiler
  
  * recent llvm/clang
  * recent gcc

* OS
  
  * ubuntu 20.04LTS
  * ubuntu 18.04LTS
  * CentOS something
  * Mac OSX whatever is easiest on Travis

* PROJ
  
  * 6.3
  * 5.[2-9].[0-9]
  * 4.9.?
  * Eventually Proj 7

* GMT
  
  * 6.x
  * 5.4.x

I think we should not have a Ubuntu 16.04LTS build. And as soon as there is an Ubuntu 20.04LTS, we should not have 19.04 or 19.10.

[schwehr]

My pref would be Ubuntu 20.04LTS Focal if that's possible. Assuming that it's not still a mess from development

[schwehr]

Very cool! But, GDAL 1? Eeek. That's really scary. We should probably put something in the MB-System configure that just says no to GDAL < 2. There were literally thousands of bugs fixed since then.

[berkowski]

I'd only ask that you add the GDAL check after making a final 5.7.6 release as we need the EM2040 capability.

At this point it seems like the major linux targets have been addressed. If someone wants to do a quick writeup of how they compile from source on OSX I can add that process too. Otherwise I think we're at the point to start the PR process.

There's still work to do, like migrating the images to a new account on hub.docker.com and creating some new API keys for the travis config, but those can be tickboxes in the PR.

[schwehr]

Awesome! Homebrew is the usual way that I have seen people do Travis Mac builds, but I have given up on building on the Mac... With an extra nudge from the government with "fedramp". I live in a varient of Debian testing now. I believe the website instructions use homebrew. I know Dave does. Can you explain why the EM2040 needs gdal 1.x?

…

On Sun, Jan 19, 2020, 6:08 AM Zac Berkowitz ***@***.***> wrote: I'd only ask that you add the GDAL check after making a final 5.7.6 release as we need the EM2040 capability. At this point it seems like the major linux targets have been addressed. If someone wants to do a quick writeup of how they compile from source on OSX I can add that process too. Otherwise I think we're at the point to start the PR process. There's still work to do, like migrating the images to a new account on hub.docker.com and creating some new API keys for the travis config, but those can be tickboxes in the PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#839?email_source=notifications&email_token=AAC7F3IMT7VEMZQYV6LHMSTQ6RNERA5CNFSM4KHJIA7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKTBTQ#issuecomment-576008398>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC7F3LM3YRU5PEZAKXPR63Q6RNERANCNFSM4KHJIA7A> .

[berkowski]

It's not that the EM2040 needs gdal 1.x, it's just that the Sentry AUV switched to the EM2040 and we're still using 16.04. There is the desire to move to a newer Ubuntu but no definite plan. Introducing the GDAL version requirement would force us to either use the last-good (for us) beta version or an unplanned update to our processing OS.

[schwehr]

Ok. Makes sense to not actively ban GDAL less than 2. However, as a maintainer of GDAL, I strongly urge you to make a plan to upgrade GDAL.

[dwcaress]

GDAL is only in the mix because it is required by GMT. MB-System does not make any GDAL calls directly, and in common usage does not trigger any use of GDAL through GMT. This could change of course, but for now what version of GDAL is there is really a GMT package manager issue.

[berkowski]

I hadn't expected #866 to be merged so quickly without migrating the Docker images to an MB-System controlled account. I don't mind keeping them there for now but I figure at some point you'll want to "own" the whole stack.

I'm going to make a small PR to ensure that docker images are only built by travis when triggered by dwcaress/MB-System and close this issue.

[berkowski]

PR mentioned above is up at #868