openorienteering / superbuild

Cross-platform superbuild of OpenOrienteering software dependencies

OpenOrienteering SuperBuild

Project goals

• Reproducible builds of packages, with clearly identified sources and scripts
• Leverage CMake language and modules
• Single central CMakeLists.txt file
• Single CMake script per package
• Handling of package dependencies
• Parallel build of packages
• Shadow build for multiple toolchains from single unpacked source directory
• Parallel build for multiple toolchains at the same time
• Leverage Debian/Ubuntu archives for cross-build and security patches
• Being friendly to IDEs and CMake GUIs

Usage

• Create a build directory.
• Run cmake SOURCE_DIR [ CONFIGURATION_OPTIONS ] from the build directory.
  • Run cmake with -DENABLE_<toolchain>=1 to enable an extra toolchain.
  • Run cmake with -DUSE_SYSTEM_<package>=0 to force the build of a package which is already provided by the system or toolchain.
  • Run cmake -L to see available configuration options.
• Run make PACKAGE-VERSION[-<toolchain>] [ -jNUM_JOBS ] from the build directory to build a package.
• Run make PACKAGE-VERSION[-<toolchain>]-package from the build directory to produce a distributable package (such as ZIP, installer, APK) for a source package.
• You may overwrite some variables for each toolchain <toolchain>_BUILD_TYPE - The CMAKE_BUILD_TYPE for this toolchain (default: CMAKE_BUILD_TYPE). <toolchain>_INSTALL_DIR - The root directory where files will be placed (default: PROJECT_BINARY_DIR//install). <toolchain>_INSTALL_PREFIX - The path where files will be located in the target system. <toolchain>_TOOLCHAIN_DIR - The root directory where the toolchain files are located.
• You may disable binary package depedencies by setting `DISABLE_DEPENDENCIES'. However, dependencies on the toolchain and on package sources will always be active.

Writing Package Files

• Package files need to be in the source directory.
• The package file name must end with ".cmake". The recommended form is "NAME-VERSION.cmake".
• The core element of a package file is the call of the superbuild_package macro. Parameters of this macro are:
  • NAME - The package name
  • VERSION - The package version and revision, max. 4 components (a.b.c.d).
  • SYSTEM_NAME - Declares the package to be the toolchain for this system.
  • DEPENDS - Names of packages this one depends one. Names may contain exact versions (DEPENDEE-a.b.c).
  • SOURCE - Download, update and patch options as understood by CMake's ExternalProject, or the name of another package, reusing that package's source.
  • SOURCE_WRITE - Write source files, given by pairs of file name and variable name. File names are interpreted relative to the SOURCE_DIR. If the named variable is a list, the items are joined, like file(WRITE list...) does.
  • USING - Names of variables which are to be passed from the top-level configuration to the configuration for a target toolchain (BUILD, BUILD_CONDITION). Use this to pass e.g. configuration options.
  • BUILD_CONDITION - optional CMake code which may set the variable BUILD_CONDITION` to false in order to disable building of the package. This may be used to check for system libraries which can be used instead of the current package.
  • BUILD - Configure, build, install and test options as understood by CMake's ExternalProject. This parameter's value shall be given quoted by double square brackets ([[ ... ]]) because the expansion of variables and generator expressions must take place in the context of the target toolchain.
  • EXECUTABLES - Declares executables in the default toolchain's build tree which shall be mirrored in the global build tree, for convenient use in IDEs (run, debug). Similar to CMake's add_executable, you need to add the keyword MACOSX_BUNDLE after executables which are application bundles on macOS.
• In addition to variables named by USING, the following variables are always available during configuration of package for a particular toolchain:
  • CMAKE_TOOLCHAIN_FILE
  • HOST_DIR
  • TOOLCHAIN_DIR
  • SOURCE_DIR
  • BINARY_DIR
  • INSTALL_DIR
  • TMP_DIR
• To apply patches from a Debian or Ubuntu modification archive (e.g. foo_1.0-1.debian.tar.gz), add a package (e.g. foo-patches 1.0-1) for this archive without build steps, make this package a dependency of the original package (e.g. foo 1.0), and add the following to the SOURCE argument of the original package:

  PATCH_COMMAND
    "${CMAKE_COMMAND}" -Dpackage=foo-patches-1.0-1 -P "${APPLY_PATCHES_SERIES}"

Writing Toolchain Package Files

• Provide an ENABLE_<TOOLCHAIN_NAME> configuration option, default 0 (OFF).
• Use the SYSTEM_NAME argument to declare the target system's name.
• Install a toolchain.cmake file.
  • Set SUPERBUILD_TOOLCHAIN_TRIPLET to the target triplet.
  • Set the following variables to adjust CMake progress information:

set(CMAKE_RULE_MESSAGES   OFF CACHE BOOL "Whether to report a message for each make rule")
set(CMAKE_TARGET_MESSAGES OFF CACHE BOOL "Whether to report a message for each target")
set(CMAKE_VERBOSE_MAKEFILE ON CACHE BOOL "Enable verbose output from Makefile builds")

TODO list

• Organization and product configuration (for deployment locations and packaging)
• Automatic collection of scripts and sources
• Automatic tests
• Cleanup

superbuild's Issues

What are the expected values for the LICENSING_COPYRIGHT_DIR and LICENSING_COMMON_DIR variables

I'm trying to build Mapper using superbuild but it fails because of the two variables above not being set, but I have no directories with the expected TXT files, not in the mapper dir nor in the superbuild dir.

What are the expected values for those variables?

Output from cmake below.

AdministorsMBP2:build administrator$ cmake Mapper_BUILD_CLIPPER=On -D PROJ4_ROOT="/usr/local/Cellar/proj/5.1.0/"  ../../mapper 
-- Configuring Mapper user manual
-- Conditional manual sections: MACOS
-- Configuring Mapper translations
-- Configuring Mapper packaging
-- Qt Assistant - found
   generic/qevdevkeyboardplugin plugin library - not found
   generic/qevdevmouseplugin plugin library - not found
   generic/qevdevtabletplugin plugin library - not found
   generic/qevdevtouchplugin plugin library - not found
   imageformats/qgif plugin library - found
   imageformats/qicns plugin library - found
   imageformats/qico plugin library - found
   imageformats/qjp2 plugin library - not found
   imageformats/qjpeg plugin library - found
   imageformats/qtiff plugin library - found
   imageformats/qwebp plugin library - found
   platforminputcontexts/composeplatforminputcontextplugin plugin library - not found
   platforminputcontexts/ibusplatforminputcontextplugin plugin library - not found
   platforms/qcocoa plugin library - found
   platforms/qwindows plugin library - not found
   platforms/qxcb plugin library - not found
   position/qtposition_cl plugin library - found
   position/qtposition_geoclue plugin library - found
   position/qtposition_serialnmea plugin library - found
   printsupport/cocoaprintersupport plugin library - found
   printsupport/cupsprintersupport plugin library - not found
   printsupport/windowsprintersupport plugin library - not found
   sqldrivers/qsqlite plugin library - found
-- Configuring Mapper licensing documentation
-- Licensing information provided by: superbuild
-- LICENSING_COPYRIGHT_DIR: LICENSING_COPYRIGHT_DIR-NOTFOUND
-- LICENSING_COMMON_DIR:    LICENSING_COMMON_DIR-NOTFOUND
CMake Error at doc/licensing/superbuild-licensing.cmake:36 (message):
  Both LICENSING_COPYRIGHT_DIR and LICENSING_COMMON_DIR must be set
Call Stack (most recent call first):
  doc/licensing/CMakeLists.txt:119 (include)

Build WebP

WebP is used in qtimageformats and in gdal.

Qt 5.12 LTS

Qt 5.12 LTS was released on Dec 6, 2018.

• Superbuild script
• Attribution
• QTBUG-71946: Fonts near-unreadable on macOS Mojave
• QTBUG-72078: Windows: Not able drop QDockWindow back to dock by drag'n'drop
• QTBUG-72101: When suspending an [Android] application after the keyboard has been shown it will hang on resuming
• QTBUG-72174: When a qch file is read-only then it will stop and fail as it cannot create index tables
• QTBUG-72291: Incorrect horizontal accuracy comparison for competing position result
• QTBUG-72848: locale de_DE.UTF-8 causes ',' instead of '.' in PDF metadata

(All regression in Qt 5.12.0 | All fixes in Qt 5.12.1)

APK build failure on openSUSE Tumbleweed and Debian Buster

Not sure this is caused by Superbuild but I'm recording my trouble here so that others can follow up.

Trying to build Android APK results in the following error.

...
Scanning dependencies of target qtbase-5.6.2-arm-linux-androideabi
[ 35%] Generating configuration for qtbase-5.6.2-arm-linux-androideabi
-- Generating qtbase-5.6.2 for system arm-linux-androideabi
-- Configuring done
CMake Error at CMakeLists.txt:32 (file):
  Error evaluating generator expression:

    $<${crosscompiling}:
        $<${windows}:
          mkdir -p "${SOURCE_DIR}/mkspecs/win32-g++-${SYSTEM_NAME}"
          COMMAND cp "${SOURCE_DIR}/mkspecs/win32-g++/qmake.conf" "${SOURCE_DIR}/mkspecs/win32-g++-${SYSTEM_NAME}/"
          COMMAND cp "${SOURCE_DIR}/mkspecs/win32-g++/qplatformdefs.h" "${SOURCE_DIR}/mkspecs/win32-g++-${SYSTEM_NAME}/"
          COMMAND sed -i -f "qmake_conf_changes.sed" "${SOURCE_DIR}/mkspecs/win32-g++-${SYSTEM_NAME}/qmake.conf"
        >
        $<${android}:
          mkdir -p "${SOURCE_DIR}/mkspecs/android-g++-${SYSTEM_NAME}"
          COMMAND cp "${SOURCE_DIR}/mkspecs/android-g++/qmake.conf" "${SOURCE_DIR}/mkspecs/android-g++-${SYSTEM_NAME}/"
          COMMAND cp "${SOURCE_DIR}/mkspecs/android-g++/qplatformdefs.h" "${SOURCE_DIR}/mkspecs/android-g++-${SYSTEM_NAME}/"
          COMMAND sed -i -f "qmake_conf_changes.sed" "${SOURCE_DIR}/mkspecs/android-g++-${SYSTEM_NAME}/qmake.conf"
        >
        COMMAND
         # Cf. qtbase configure "SYSTEM_VARIABLES"
         "${CMAKE_COMMAND}" -E env 
           --unset=AR
           --unset=RANLIB
           --unset=STRIP
           --unset=OBJDUMP
           --unset=LD
           --unset=CC
           --unset=CXX
           --unset=CFLAGS
           --unset=CXXFLAGS
           --unset=LDFLAGS
           # fall through
      >

  Expression did not evaluate to a known generator expression


-- Generating done
-- Build files have been written to: /data/build/android-build/arm-linux-androideabi/qtbase-5.6.2/tmp.superbuild/config/generator
make[3]: *** [CMakeFiles/qtbase-5.6.2-arm-linux-androideabi.dir/build.make:67: arm-linux-androideabi/qtbase-5.6.2/tmp.superbuild/CMakeLists.txt] Error 1
make[3]: *** Deleting file 'arm-linux-androideabi/qtbase-5.6.2/tmp.superbuild/CMakeLists.txt'
make[2]: *** [CMakeFiles/Makefile2:1797: CMakeFiles/qtbase-5.6.2-arm-linux-androideabi.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:4946: CMakeFiles/openorienteering-mapper-master-arm-linux-androideabi-package.dir/rule] Error 2
make: *** [Makefile:1990: openorienteering-mapper-master-arm-linux-androideabi-package] Error 2
lpechacek@fmn:/data/build/android-build> 

There has been various trouble with Superbuild on Tumbleweed but this one is completely new.

Steps to reproduce:

1. cmake -D ENABLE_arm-linux-androideabi:BOOL=ON /data/src/superbuild/
2. make openorienteering-mapper-master-arm-linux-androideabi-package

On Debian Buster JAVA_HOME has to point to OpenJDK 8 so that Android sdk_manager works.

Superbuild version: v20190622.4
Mapper-master version: 88576a0784c7079f4a94cc9c47207ee16741908f

Update MSYS2 in CI

• The currently cached MSYS2 configuration cannot restored by a fresh download, due to significant changes in MSYS2 setup.
• There are new generic instructions for CI setup: https://www.msys2.org/docs/ci/#other-systems.

GDAL 2.2.4 is in Debian unstable

https://tracker.debian.org/pkg/gdal

Update doxygen

Newer versions of doxygen are available, even packaged for Debian (https://tracker.debian.org/pkg/doxygen).

However, updating introduces new dependencies. An attempt to update from 1.8.13 to 1.8.16 failed in CI on macOS 10.14 due to an outdated bison:

-- Found BISON: /usr/bin/bison (found version "2.3") 
   Doxygen requires at least bison version 2.7 (installed: 2.3)

Alternatively, doxygen might be replaced by another tool, e.g. Pandoc.

PROJ 5.0.0 released

http://lists.maptools.org/pipermail/proj/2018-March/008098.html

Build openJPEG

Needed for JPEG2000 support via GDAL.
License: BSD-2
https://tracker.debian.org/pkg/openjpeg2

qtbase-5.6.2 requires libxcb

The configure step of qtbase tests linking against system libxcb. However, libxcb is not included in the superbuild package set. Is that intended or are there parts of superbuild missing in this repo?

Qt 5.9.2 is in Debian unstable

https://tracker.debian.org/pkg/qtbase-opensource-src

Builds variants: "stable", "unstable", "master"

I see that week ago you start build openorienteering-mapper-master_* binary packages.

• http://download.opensuse.org/repositories/home:/dg0yt/Ubuntu_16.04/amd64/
  • stable - openorienteering-mapper_0.7.0-0+3.1_amd64.deb
  • unstable - openorienteering-mapper-unstable_0.8.20171201-0+473.1_amd64.deb
  • master - openorienteering-mapper-master_0.8.20171211-0+22.1_amd64.deb

So, what status of openorienteering-mapper-unstable_* (that not updated now) builds are?

Build NASM for macOS

NASM is recommended for libjpeg-turbo x64.

Update giflib

The current package version is 5.1.4-3, the version which is in Debian stable at the moment.

A new version of giflib is available. However, it dropped the build configuration system. Now, it is hard to cross-compile (Android, MinGW), and not possible to build out-of-source.

Besides updating, one alternative is to not build and use the giflib package: Qt uses a different implementation, WebP needs it only for some command line tools, and we don't really need in GDAL if it works in Qt.

GDAL itself has a (modified) copy of giflib 4.1.6.