cameronfyfe / hashdeps

GNU make makefile for remaking targets only when the contents of dependencies (file hashes) change rather than modification times

Hashdeps

GNU make file which can be included to configure rebuilding of a target based on a dependency's content changing rather than its modification time.

A good parallel to draw is with ccache, except this works for all build targets, not just C files.

Use Cases

GNU make decides that a target needs rebuilding if a dependency is 'newer' than the target file. It does this by comparing the modification timestamp of the target and dependency files. However, these timestamps can often change such that make decides a target needs rebuilding when actually nothing has changed. Here are some examples.

Changing Git Branches in a Local Clone

In a git codebase, you:

• build your code
• switch branches, which changes a file's contents, but don't build anything
• checkout the original branch
• build again with the code exactly as it was before.

Since Git commits don't include file timestamps, the checkout of the original branch sets the modification times of any changed files to the current time, so the build through make will still trigger rebuilds.

CI System Caching/Passing Built Objects Between Instances

Suppose you have a CI system that builds objects and can cache the objects between builds of the same type, or passes a partially built source tree between stages. Each time the CI system starts a build, the source files may have been re-checked out from version control, so may have newer modification times on disk than the built files. In this case, it's preferable that make checks the content of the source files is the same as when the objects were built, rather than file modification times.

Requirements

• Should have no requirement on the version of GNU make.
• Requires the md5sum utility to be installed - installable as coreutils in package managers.
• Only has Linux support currently.

Usage

1. Add the makefile to your project.

   1. Either add this repository as a git submodule of your Git project, or
   2. just download the hashdeps.mk file into a suitable location in your project.

2. Include hashdeps.mk from your main Makefile - e.g. assuming you put the file in a directory makefiles/, add the line:

   include makefiles/hashdeps.mk

   Because hashdeps.mk defines values that you then reference in your own make rules, it must be included in the process as soon as possible - i.e. at the very top of the main Makefile.

3. Wrap any dependencies you want to be hashed in a make function call to hash_deps, and wrap any references to uses of the built in variables providing the dependency names in recipes in unhash_deps - see the simple examples below.

There's also built in support for GCC's automatic dependency generation allowing you to easily hash dependencies only referenced in these autogenerated files using hash_deps_in_autogen_dep_file. More detailed information and an example is covered at the top of hashdeps.mk.

More Information on Usage

• This utility takes the md5sum of dependencies to determine if they have changed, and should be sufficiently unique for most use cases. This can easily be replaced with e.g. sha256sum via a simple config option if desired, but there's nothing cryptographically secure about this tool.

• While this utility helps speed up build times in the main uses cases covered above, in completely clean builds there will be the overhead of computing hashes on top of any usual building work and so these will almost certainly be some amount slower.

• The default use case for this utility is in preventing rebuilds when source files have new timestamps, but their content is unchanged. However there is a mode of operation where the hashes are always computed and checked, e.g. you need to cope with source files changing but still having old timestamps and ensuring a rebuild is still triggered - see the detailed information on this in hashdeps.mk.

Configuration

Users can set various configuration variables - e.g. by setting the variables before including the provided makefile, or at the command line call to make. For example:

• A simple flag to disable this utility from doing anything.
• Change the filenames used for storing file hashes, and storing them separate from source files.

...and more. All configuration variables are documented at the start of the hashdeps.mk file to keep the docs in one location.

Simple Examples

See the unit test files for other examples of this utility in use.

Converting a Target to use Hashed Dependencies

Starting with:

combined.txt: a.txt b.txt
    echo "Concatenating files"
    cat $^ > $@
    # The make syntax for:
    # cat a.txt b.txt > combined.txt

All that needs to be done is include the makefile and pass the dependencies to hash to the hash_deps function.

include hashdeps.mk

# This file is only regenerated if the contents of a.txt or b.txt changes.
# e.g. running:
# 'make combined.txt; touch a.txt; make combined.txt'
# only echo-es once, the first time.
combined.txt: $(call hash_deps,a.txt b.txt)
    echo "Concatenating files"
    cat $(call unhash_deps,$^) > $@

Development

• Install shunit2 using your system's package manager.
• See instructions here to get the latest shellcheck - typically the one in package managers is older and doesn't report all issues.

Run tests with make test