GMT 6.x.x and MB-System issues about mb-system

[dwcaress]

Paul, Thanks for getting into this. First, the changes to the codebase since the last major release (5.7.5) are so large that I don’t think it’s worth trying to identify fixes relative to that release. Either work with the current state of the master branch or the most recent beta release (5.7.6beta31). I’m trying to get to the next beta release, but I have to complete some format support update work for Kongsberg and Teledyne data. Second, all of my testing on MacOs is with the prerequisites installed with HomeBrew. I haven’t used Fink for several years and I haven’t looked at MacPorts in over a decade. I’m not shocked if there are problems with a MacPorts based installation. My wife’s laptop is pristine (no Xcode, no GMT or MB), so I’ll try to set up a MacPorts based install on it for testing. I’ve attempted to update the prerequisites list in the website and the README.md file. I still have to do the equivalent edits to the old man pages. The GMT related preprocessor commands you cite have changed since 5.7.5, and are working in my test builds on MacOs High Sierra and Catalina, and on Ubuntu 16, 18, 19 and on CentOs 7 with GMT releases 5.4 and 6.0. I may still have things wrong, but please look at the current code. I would like to just require GMT 6 and later (and PROJ 6.1 or later), but I’m not sure we can get away with that now. Cheers, Dave

…

On Apr 11, 2020, at 1:52 PM, Paul Wessel ***@***.***> wrote: Hi MB-System developers. The GMT developers have decided to be more proactive in making sure MB-system works well with GMT 6.x and future releases. I have therefore forked your repo and trying to build and fix anything so I can submit a pull request. This is on macOS Catalina with macports. Since I rarely build MB it is likely I am running into trivial issues but they are preventing me from doing my testing. Here is my status: • Your list of prerequisites seems incomplete: proj is not mentioned, and GDAL is missing from your README.md (but mentioned on the MBARI list). • I have a configure script that has worked in the past, and works (up to a point) with 5.7.5. The problem is that despite configuring with --with-proj-lib=/opt/local/lib/proj6/lib --with-proj-include=/opt/local/lib/proj6/include, the master build fails with g++ -std=gnu++11 -DHAVE_CONFIG_H -I. -I../../src/mbio -I../../src/mbio -I../../src/mbaux -I../../src/gsf -I/Users/pwessel/GMTdev/gmt-dev/dbuild/gmt6/include/gmt -I/opt/local/include -I/opt/local/include -g -O2 -MT mbsvpselect.o -MD -MP -MF $depbase.Tpo -c -o mbsvpselect.o mbsvpselect.cc &&\ mv -f $depbase.Tpo $depbase.Po mbsvpselect.cc:193:10: fatal error: 'geodesic.h' file not found #include <geodesic.h> That file exists, so I am not sure why my includes are not honored for this makefile. I see there is now lots of C++ files that have been added so perhaps the build for these are not robust yet. So I try the 5.7.5 tarball instead. This works until we hit src/gmt. This step fail because the precompiler directives to check for GMT 5 versions fail on 6.0.0 and think it is < 5.2.1, and chaos ensues. Specifically, #if GMT_MINOR_VERSION == 1 && GMT_RELEASE_VERSION < 2 if (gmt_check_filearg(GMT, '<', opt->arg, GMT_IN)) { #else if (gmt_check_filearg(GMT, '<', opt->arg, GMT_IN, GMT_IS_DATASET)) { #endif needs to be #if GMT_MAJOR_VERSION == 5 && GMT_MINOR_VERSION == 1 && GMT_RELEASE_VERSION < 2 if (gmt_check_filearg(GMT, '<', opt->arg, GMT_IN)) { #else if (gmt_check_filearg(GMT, '<', opt->arg, GMT_IN, GMT_IS_DATASET)) { #endif Same for mbcontour.c and mbgrdtiff.c. Similarly, the two latter tools also need #if GMT_MAJOR_VERSION == 6 || (GMT_MAJOR_VERSION == 5 && GMT_MINOR_VERSION > 3) n_errors += gmt_M_check_condition(GMT, !GMT->common.R.active[RSET], "Syntax error: Must specify -R option\n"); #else n_errors += gmt_M_check_condition(GMT, !GMT->common.R.active, "Syntax error: Must specify -R option\n"); #endif to compile. With these edits, mb-system builds with GMT 6.x. I note your instructions state you require GMT 5.4 or later. If so, you could have configure check for that (via gmt --version) and stop if that is not true - you could then remove all those #ifdefs that deal with things older than 5.4. In fact, you have lots of defines at the top of each of these three C codes that deal with 5.2 and older. You could yank all that if you stick to your 5.4 cutoff. Even Ubuntu seems to have managed to move up to 5.4.5. However, if you can address the build issues with g++ above (or tell me how to circumvent it) then I am happy to submit a pull request. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[joa-quim]

Dave, just let me add that I'm sad that you decided to go into the C++ way. C++ is simply no good neighbor. Now the possibility of having the MB programs adapted to be GMT modules. like mbcontour for example, and hence be easily accessible from other languages like Matlab or Julia is definitively gone.

[dwcaress]

Joaquim, At least for the foreseeable future, the libraries all remain as C. Most of the command line utilities that have been turned into *.cc are only really changed in which system header files are included. All new graphical programs will be C++, but will call the usual C libraries. I don’t think there’s anything precluding you from making a version of a utility like mbgrid that is structured as a GMT module. The code is still really C that just happens to be compiled as C++. I’m not going to do that, but I don’t mind if you or anyone else takes that on. Have you solved the problem of embedding the coordinate system definition properly in tiff images from grdimage to generate full GeoTiffs? As we discussed last summer, that will make mbgrdtiff obsolete. Cheers, Dave

…

On Apr 12, 2020, at 2:55 AM, Joaquim ***@***.***> wrote: Dave, just let me add that I'm sad that you decided to go into the C++ way. C++ is simply no good neighbor. Now the possibility of having the MB programs adapted to be GMT modules. like mbcontour for example, and hence be easily accessible from other languages like Matlab or Julia is definitively gone. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[joa-quim]

Dave,

All GMT modules end up as shared libraries. Shared libraries in C++ suffer from the name mangling issue and are not callable by other languages (not even C). That is why C++ is no good neighbor.

Regarding the GeoTiffs, sorry I don't get. grdimage does that since the option to write GeoTiffs was added many years ago.

[PaulWessel]

I think Dave is particular about what is written to the geotiff. THe easy solution would be to rip out most of mbgrdtiff and simply do a module call to GMT_grdimage and return a GMT_IMAGE, which mbgrdtiff can then write out as it sees fit.

[dwcaress]

Correct. At the GMT Summit last summer we did a test to see if a tiff produced by GMT had the cooordinate reference system embedded in compliance with the GeoTiff standard, and it did not. This explains why tiff’s produced through the method Joaquim has suggested can’t actually be imported directly into ArcGIS, ERMapper, Winfrog, etc. Generating proper GeoTiff’s that are generally portable is the purpose of mbgrdtiff. I was just inquiring if that issue had been addressed. From my point of view, the primary benefit of generating portable GeoTiff’s from GMT would be the ability to build GeoTiffs with contours, annotations,graticules, etc - mbgrdtiff just produces a simple colored and shaded image. This is fine for pure GIS applications where users display as many layers as they want, but it is lacking for realtime navigation applications like Winfrog that only display a single layer.

[joa-quim]

Ok, this is a simple example with geotiff grids. Tried to project it to make it more clear but some shit in propagating the proj coords metadata is happening

grdimage does not create geotiffs with other than the image. I.e., not images with vector lines. But psconvert does it.

grdmath -R0/20/30/50 -I0.1 X Y MUL = lixo.grd
grdedit lixo.grd -J+proj=longlat
grdconvert lixo.grd lixo.tif=gd:GTiff
gdalinfo lixo.tif

Driver: GTiff/GeoTIFF
Files: lixo.tif
Size is 201, 201
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
...

[dwcaress]

Joaquim, What you have below tests setting the CRS of a scalar grid then converted directly to TIFF format. That is different than producing a shaded relief RGB image with grdimage, and having the CRS embedded in that image according to the GeoTiff standard. I will try again to see what is in a Tiff file generated directly by grdimage. However, in our brief test last summer you and I found that the CRS encoding just wasn’t there. Cheers, Dave

…

On Apr 12, 2020, at 4:09 PM, Joaquim ***@***.***> wrote: Ok, this is a simple example with geotiff grids. Tried to project it to make it more clear but some shit in propagating the proj coords metadata is happening grdimage does not create geotiffs with other than the image. I.e., not images with vector lines. But psconvert does it. grdmath -R0/20/30/50 -I0.1 X Y MUL = lixo.grd grdedit lixo.grd -J+proj=longlat grdconvert lixo.grd lixo.tif=gd:GTiff gdalinfo lixo.tif Driver: GTiff/GeoTIFF Files: lixo.tif Size is 201, 201 Coordinate System is: GEOGCRS["WGS 84", DATUM["World Geodetic System 1984", ELLIPSOID["WGS 84",6378137,298.257223563, LENGTHUNIT["metre",1]]], ... — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[joa-quim]

Dave, you'll excuse me but I don't remember that we have done that failed test. I will be making a PR in GMT to fix some bugs related to the non-propagation in some cases of the CRS info when writing either nc or GTiff files. But that is a separate issue. With those fixes one can do
(But maybe it will work for you because the bugs occurred when one uses the -J+proj=utm+zone=32 and the necessary space before +zone was not introduced in some part of the code)

gmt grdmath -R0/20/30/50 -I0.1 X Y MUL = lixo.grd
gmt grdproject lixo.grd -J"+proj=utm +zone=32" -Glixo_utm.grd
gmt grdconvert lixo_utm.grd lixo_utm.tiff=gd:GTiff

gmt grdimage lixo_utm.grd  -JX8c -I+d -Ba -BWSen -P -K > gimg_tif.ps
echo 10 40 | mapproject -J"+proj=utm +zone=32" | gmt psxy -JX8c -Rlixo_utm.grd -Sc1c -Gwhite -O -K >> gimg_tif.ps
gmt grdimage lixo_utm.tiff -JX8c -I+d -Ba -BwSen -X8.5c -O -K >> gimg_tif.ps
echo 10 40 | mapproject -J"+proj=utm +zone=32" | gmt psxy -JX8 -Rlixo_utm.tiff -Sc1c -Gwhite -O -K >> gimg_tif.ps

gmt grdimage lixo_utm.grd -I+d -Agrdimg.tiff        # This is a shaded GTiff IMAGE, not a GRID

gmt grdimage grdimg.tiff -JX8c -X-8.5c -Y8.5c -Ba -BWseN -O -K >> gimg_tif.ps
echo 10 40 | gmt mapproject -J"+proj=utm +zone=32" | gmt psxy -JX8c -Rgrdimg.tiff -Sc1c -Gwhite -O >> gimg_tif.ps

and get

[joa-quim]

PR submitted in GMT. But with this we hijacked the original issue. Sorry.

[PaulWessel]

BTW, I am this close to having redone mbgrdtiff via a call to GMT_grdimage.

[PaulWessel]

I have hacked up a new version of mbgrdtiff.c that uses a call to GMT_grdimage to do the work. Since I am unable to use your github version directly [there must be something that is brew-specific about the configure process], I did this with 5.7.5. Since the new version adds a few new things and deletes most of the old grdimage-lookalike code I think you can easily do this with what is attached and insert into your master version. Here are the steps:

1. The main differences is a new static function called mbgrdtiff_get_image that takes your options and builds a command string and then calls grdimage and receives the image. Just insert that code into your mbgrdtiff.c above the main function.
2. Instead of the old image calculations, just keep the new call to that new function and the next three lines, i.e., everything between

/*---------------------------- This is the mbgrdtiff main code ----------------------------*/
and
/*------------------------- Write out the GeoTiff and world files -------------------------*/

3. Compile with -Wall and remove all the unused variables. I tested this and both the new and old 5.7.5 mbgrdtiff gives me the same tiff file.

One caveat is that this requires GMT 6.x since I had to fix some issues in grdimage to make this work.

mbgrdtiff_new.c.txt

[PaulWessel]

Note the old mbgrdtiff lists options in the usage message that are not available in that module (-B, -K, -P, -X, etc). I deleted all of those.
Also, looks like 5.7.5 version crashes since it is accessing Grid_orig[0] in a fprintf (stderr message long after Grid_orig has been destroyed.

[dwcaress]

Ok, thanks. I will try it out. Jumping back to the beginning of the thread, I've duplicated the build error you found when building with prerequisites installed through MacPorts.

[dwcaress]

And I fixed the build error that died at mbsvpselect. I needed to add libproj_CPPFLAGS to AM_CPPFLAGS in src/utilities/Makefile.am, and then resconstruct the build system. This fix has been merged into master.

[PaulWessel]

Great, I will give it a try. if that works, would you want me to submit a PR for the tiff stuff instead?

[dwcaress]

Does this work with the GMT 6.0.0 release?

[PaulWessel]

No, since it doing this I found a bug. This was the first time we called GMT_grdimage and wanted a memory reference to an image back, so had not been tested before. Minor bug, but not in 6.0.0.

[dwcaress]

So, no, don't put it into a PR. You've given me the code - I need to wrap it in conditionals so we can still build and work with GMT 6.0.0, and for the time being, GMT 5.4.5. I'll try to do it moderately cleanly. I take it the required mods to GMT are merged into your master branch, correct?

Thanks. This is definitely an improvement.

[PaulWessel]

OK, that is fine. Yes, the fix has ben committed to our master branch. Let me know if anything fails.

[dwcaress]

I’ve now successfully built MB-System on a Mac running Mojave that has all prerequisites installed using MacPorts. This pertains to the current state of the master branch on Github, and excludes building the TRN utilities, which are still problematic. Here are my notes from tonights successful installation. This laptop began without XCode or MacPorts installed. I largely followed the directions at: https://www.macports.org/install.php (1) Install XCode and XCode command line tools (2) Download the MacPorts package installer for the relevant version of MacOs. After installation, run: sudo port -v selfupdate (3) Use the port command to install the necessary prerequisites: sudo port install gmt6 sudo port install proj6 sudo port install fftw-3 sudo port install xorg sudo port install openmotif sudo port install gedit (4) Deal with Python, which is a bit complication. MB-System uses Python3.X and assumes it is available as a command called python3. MacPorts can install many versions of a package simultaneously. In the case of python, this takes the form of separate packages for different Python versions. You can list the installed versions using the select command of port. After installing GMT, proj, and gedit, we find that both python27 and python36 have been installed, and of course Apple’s default python27 is available as /usr/bin/python: port select - -list python none (active) python27 python27-apple python36 We want to use Python 3.8, so we install that: sudo port install python38 Now, python38 shows up in the listing: port select --list python none (active) python27 python27-apple python36 python38 We need a command named python3 to show up in our path. MacPorts has a facility to create a custom command linked to another command: port select --set python3 python38 (5) Install MB-System Download the MB-System source package. I downloaded the master branch of the MB-System repository on Github. Run configure: LDFLAGS="-L/opt/X11/lib" \ CFLAGS="-g -I/opt/X11/include" \ ./configure \ --prefix=/usr/local \ --with-proj-include=/opt/local/lib/proj6/include \ --with-proj-lib=/opt/local/lib/proj6/lib \ --with-otps-dir=/usr/local/opt/otps \ --enable-hardening \ --enable-test make make check sudo make install This worked for me… Cheers, Dave

…

On Apr 13, 2020, at 10:34 PM, Paul Wessel ***@***.***> wrote: OK, that is fine. Yes, the fix has ben committed to our master branch. Let me know if anything fails. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

---------------------------------------------------- David W. Caress Principal Engineer Seafloor Mapping Lab Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 [email protected] http://www.mbari.org/~caress/ Phone: [phone]

[schwehr]

Dave, just let me add that I'm sad that you decided to go into the C++ way. C++ is simply no good neighbor. Now the possibility of having the MB programs adapted to be GMT modules. like mbcontour for example, and hence be easily accessible from other languages like Matlab or Julia is definitively gone.

I strongly disagree with your blanket statement about C++.

C++ doesn't impact the ABI if used from a C API. e.g. gdal and lots of other open source projects. IMHO, C++ ( > 11 ) makes for much more robust internals to libraries. I did nothing with C++ in MB-System that impacts the ABI. If it was up to me, all of MB-System would be C++ >= 17 with a C APIs for basic use by others. The reliability and standardization of basic functionality is such a huge win.