freecad / freecad-ports-cache

FreeCAD ports cache to reduce Travis-CI build times for Mac OS X

FreeCAD Ports Cache

Overview

This repo hosts the FreeCAD ports cache used to significantly reduce the FreeCAD Travis-CI build time on MacOS X.

Motivation and Approach

Travis-CI limits Open Source build times to 50 minutes total per build. FreeCAD, being a cross-platform QT application, requires installing Unix ports that consume nearly half of the availalble build time just to satisfy the pre-requisites causing frequent build timeouts. In order to significantly reduce the build cycle time, the ports dependencies are pre-computed and deployed to GitHub as a cache archive for use during continuous integration builds. Early tests show that a ports cache can be restored in just 3 to 4 minutes reducing the OSX build times by 20-25 minutes per cycle. This enables reliable continuous integration builds for Mac OS X for all users and ensures that FreeCAD is a good Travis-CI citizen (use shared resources wisely).

The ports cache itself is maintained using Travis-CI to pre-compute, build and deploy the cache as a GitHub release asset. FreeCAD continuous integration builds can optionally fetch and restore a ports cache before building FreeCAD by defining a single environment variable OSX_PORTS_CACHE in their respecitive Travis-CI configuration file and use convenience functions to simplify cache creation, deployment and usage from Travis.

Port cache archives are deployed on the FreeCAD ports cache repo under a release tag that corresponds to a specific FreeCAD version. The cache is identitfied by a cache descriptor of the form package_manager-pm_version-os_version-arch-build_date that is used to fetch and restore a suitable ports cache for the given build environment. The combination of these two descriminators enable caching ports for specific package managers on various operating systems and architectures for any given FreeCAD release (currently only the Homebrew package manager is recommended and supported). For example, the port cache archive named homebrew-0.9.5-10.9.5-x86_64-2016.03.21.tgz deployed under release 0.16 on the FreeCAD-ports-cache GitHub repo is a FreeCAD 0.16 port cache for Homebrew 0.9.5 running on a 64-bit architecture under OS X 10.9.5 (Mavericks). The 2016.03.21 suffix is merely a referential date; the helper functions will always fetch the latest cache file that satisfies the build environment requirements.

Ports Cache Build Status

Master	0.16	0.17

Installation and Usage

In order to use a ports cache, one must first be pre-computed (i.e. built) and deployed to GitHub. The ports cache travis config file will fully initialize (i.e. bootstrap) a ports cache repo that contains nothing more than the config file itself. Once available, clients can use a short list of convenience functions to employ am appropriate ports cache during a Travis CI build.

Cache Usage

To use an existing cache, only two convenience functions need be called after instantiating them in the current Travis shell context. One function is used to set a cache context and the other to prime the local ports cache using the cache context. Below, are the relevant excerpts that shows the cache usage.

before_install:
- |
# 1. Fetch helper functions and instantiate them in the Travis shell context
eval "$(curl -fsSL "https://raw.githubusercontent.com/${OSX_PORTS_CACHE}/travis-helpers/travis-helpers.sh")"

# 2. Create a helper cache context used by the convenience functions.  auth_token is only required if you are deploying a new cache to GitHub (see Deploying a Cache below)
cacheContext=$(create_helper_context repo="${OSX_PORTS_CACHE}" auth_token=${GH_TOKEN} release=${FREECAD_RELEASE})

# 3. Fetch the latest cache that matches the build requirement
prime_local_ports_cache $cacheContext

In FreeCAD, the travis config file controls the cache usage by setting the OSX_PORTS_CACHE environment variable that refers to the ports cache repo. If the OSX_PORTS_CACHE variable is set, then the referenced ports cache repo will be used.

NOTE: The travis-helpers.sh may be moved to the main FreeCAD repo (decision pending)

Deploying a Cache

When forking the FreeCAD-ports-cache repo for development, you will need to create a GitHUB OAUTH token with deployment privileges and assign it to a secure Travis Environment variable via the Travis-CI Settings for use within the forked ports cache's travis config file. Currently the environment variable used is GH_TOKEN. This is required for the gitHub_deploy_asset_to_release_named convenience function. Below is an excerpt from the .travis.yml file for creating and deploying a new cache archive.

GH_TOKEN=[secure]

script:
- |
# 1.  Create an archive of the installed ports with a cache descriptor for the current build environment
 portsArchive=$(create_ports_cache_archive /usr/local ${TRAVIS_BUILD_DIR}/$(ports_archive_filename))
 
 #2.  Deploy the cache archive to GitHub for the appropriate FreeCAD release
gitHub_deploy_asset_to_release_named $cacheContext $portsArchive ${FREECAD_RELEASE}

Cache Maintenance

The cache is designed to be nearly self-maintaining however under some circumstances, the following manual maintenance steps are required to refresh and deploy new version of the ports cache. Furthermore, if the cache is not refreshed, then the cache benefits should gracefully degrade from missing or outdated ports available the cache that will need to be rebuilt during the FreeCAD process (e.g. port changes during a FreeCAD release version development cycle) to no cache available (e.g. port cache not created for new FreeCAD version development or new OS X).

• Port Dependency Changes

  • Updates to Homebrew formulae or Build Environment

    If a Homebrew formula is updated or the Travis environment changes, a refreshed cache can easily be built and deployed by simply restarting the last Travis-CI job.
    

  • Changed FreeCAD port dependency

    Whenever a port dependency is added to the [FreeCAD .travis.yml config file][freecad-config] the change must also be incorporated into to the [Ports Cache .travis.yml config file](.travis.yml).  Pushing the updated .travis.yml file to the FreeCAD-ports-cache repo on GitHub will start a Travis-CI job that will build and deploy the new ports cache archive.
    

• New FreeCAD Release

  When the development cycle commences for a new version of FreeCAD commences (e.g. 0.16 release -> 0.17 development), the corresponding release identifier must be created for the FreeCAD-ports-cache repo and defined in the .travis.yml configuration file to create the initial cache.

Repo Branching Model

GitFlow like

Futures

Longer term, we will develop a FreeCAD_dependencies homebrew forumla to install the FreeCAD port dependencies. The FreeCAD_dependencies formula will be a single ports dependency definition that both the FreeCAD cache and FreeCAD CI builds will use.

Ports Cache Repo Maintainers

@ianrrees, @peterl94, @sgrogan, @wwmeyer, @yorik, @bblacey

freecad-ports-cache's Issues

Rebase ports-cache for forthcoming Travis-ci infrastructure changes

Currently our Travis builds run on macOS 10.9 (Mavericks) using Xcode 6.1 however the Travis team plans to deprecate that image on 10/31 and change the default image to mac OS 10.10 (El Capitan) using Xcode 7.3. Travis will continue to support macOS 10.10 (Yosemite) using Xcode 6.4 which is the oldest macOS supported by the homebrew package manager and Apple.

The scope of this issue is to rebase the ports cache for macOS 10.10/Xcode 6.4 and possibly macOS 10.11/Xcode 7.3 because that combination is the default and receiving the greatest userland updates that may improve build robustness and performance. Final decision pending Travis infrastructure rollout to evaluate available SDKs.

Ports-cache descriptor should reference homebrew major.minor version

Currently the ports cache descriptor uses the homebrew version as reported by brew. The homebrew team recently release homebrew 1.0.0 and in doing so, homebrew versions may include additional semantic versioning information such as the git commit sha. The results in ports-cache misses that results in longer travis build times exacerbated by the fact that homebrew now autoupdates everytime the brew install is invoked.

Reducing the ports-cache descriptor to use the homebrew major.minor revision or even major.minor.patch version will increase suitable-cache hits in our travis CI builds.

Stability Issues with the Solver Messages pane

I had an issue where the solver would say that all of my constraints were redundant. All but one or two. It would allow me to select and delete them, but immediately after it was "under constrained sketch with 40 degrees of freedom." The sketch I was working on was a relatively simple sketch. I could not figure out how to mitigate the problem so it would stop showing all of my constraints as redundant, and I had not yet saved the sketch or project, as I was just practicing, so i found no way to go back to a previous version to see if that would help the problem. If there are any log files or other needed files, I am happy to provide them.

Debug libraries are outdated within archive for VC12

I don't know if this repository is intended to be used apart from CI (I've not found if FreeCAD project manages FreeCADLibs updates),
but trying to build FreeCAD 0.17 with FreeCADLibs_11.10_x64_VC12.7z I've observed that Debug versions of (at least) OCCT libraries within archive are outdated (files are dated by '2015) - leading to linkage errors (Release version is fine).

So, if Debug libraries are not intended to be used, it might be reasonable removing them from archives, or at least indicated on github page about this limitation.

Disable cache deployment for pull requests

Travis-CI, for security reasons, refuses to set secure repository environment variables. Furthermore, it doesn't really make sense to attempt to deploy a cache asset for pull requests.

Add netgen/nglib dependency to ports-cache

Need to add the following port dependency:

brew install --with-oce ianrrees/tapdev/nglib

Once @ianrrees's pending request for the netgen/glib formula to be accepted into home-brew/science, this dependency will need to change accordingly.

Create home-brew formula to install FreeCAD port dependencies

This will provide a single definition of the FreeCAD port dependencies for both the continuous integration builds and the cache reducing the potential for divergence between the two.

Add 3DConnexion Frameworks to cache

To further reduce CI build times, the 3DConnexion frameworks should be added to the ports-cache. The FreeCAD .travis.yml can check to see if they exist (i.e. using cached ports) and if not, then install them.

Support deploying release assets by release tag

Currently, the travis-helper scripts allow deployment of release assets to a GitHub release with a given name/title. The original impetus was that release names are resilient to the underlying tag name being changed and this works well for continuous integration of builds on master.

However, when a tag is committed, the travis job should deploy the release asset to a release with that tag. In the current incarnation, the asset is deployed to the DEPLOY_NAME release and not the release with the associated tag.

Trap ERR for multi-line yaml commands in Travis config

Travis executes the build configuration declared in yaml syntax one command at a time and traps non-zero exit status. Currently, the FreeCAD travis.yaml uses yaml 3 extensions to declare multiline command in a more readable format with proper use of whitespace. It appears that multiple commands spanning a single travis yaml command do not trap ERR and propagate the ERR to the travis job processor. This may be as simple as adding the following to the travis helpers and then invoking it at the beginning of a multi-command yaml command block.

+alias set_return_on_ERR='trap "rv=\$?; echo Error occurred while executing \$BASH_COMMAND; return \$rv" ERR' set_return_on_ERR

Reliable port-caching for GitHub pull-requests from forked repos

One of the key goals for the FreeCAD ports caching strategy, is to reduce the Travis CI build times on OS X sufficiently that we can build every pull-request, especially from forked repos, as a pre-merge validation check to reduce the likelihood of a merge causing a build failure on master. The ports-caching strategy is solid and reduces the continuous integration cycle times to well below the Travis-CI Open Source 50 minute limit on OS X.

However, the port caching strategy uses the GitHub APIs to locate the most appropriate ports cache using a port cache descriptor for the given environment. The GitHub API enforces a rate limit of 60 unauthenticated requests per hour and 5000 authenticated requests per hour. With the Travis-CI infrastructure being shared across many users/builds, the unauthenticated rate limit is routinely hit so the caching strategy uses authenticated requests employing a GitHub OAUTH token set on a per repo basis. The GitHub OAUTH token is defined as a secure environment variable GH_TOKEN using the Travis-CI repo settings UI (i.e. users do not need to tailor the .travis.yml file because the token is explicitly defined outside the .travis.yml file for portability).

Unfortunately, for this use case, Travis-CI does not set secure environment variables for pull requests from forked repositories to ensure the secure variables are not compromised hence there is no way directly supported way to authenticate against the GitHub API. This means that pull requests from forked repos will almost always hit the 60 requests/minute causing the search for a matching port cache to fail that in turn will cause pull requests to build without caching undermining the core value proposition of the ports-caching strategy, and worse, causing pre-merge pull request builds to timeout.

There are at least 3 possible options:

1. Manage encrypted OAUTH tokens external from Travis
2. Use the GitHub Search API that has different rate limits(?)
3. Use direct archive URLs to download the cache (i.e. avoid indirection through the GitHub API to locate a matching cache's download URL).

Recently GitHub changed their rate limit API policy for direct archive downloads making option 3 above viable. The only real downside is that we will have a less robust cache search mechanism but in reality, it shouldn't have any tangible affect on the overarching goal for the FreeCAD continuous integration strategy.