edvardm / demake

Helper to translate Makefile rules to PyInvoke tasks.py file

Demake

Version 0.1.6

Overview

Helper to convert Makefiles to PyInvoke files, see sample run below

Rationale

Makefiles typically contain rules to build files based on dependencies, whereas PyInvoke is used for task automation. These are two separate things.

GNU Make is a powerful tool, so it is often used for task automation as well, mainly because it is already present in many development environments, and most people are familiar with the basic syntax. Problem is, Make is not very suitable for complex rules. Error handling, condition logic, passing arbitrary command line arguments etc is not where it really shines.

In the end there shouldn't even be battle between Make and task automation tools. In fact, in some cases it makes totally sense to have both PyInvoke for convenient task automation, and Make for incremental builds. PyInvoke, like many task automation tools, doesn't offer support for building items automatically based on extension, or running build targets based on file modification time. In such cases, it would be totally reasonable to use PyInvoke as main automation tool for developers, and let Make handle incremental builds and nothing else.

Sure, complex Make tasks can be extracted to shell scripts, but then the same arguments about awkward programmability would arise as with Make.

Purpose of Demake

It helps you by transforming some of the rules in existing Makefiles to tasks.py files used by PyInvoke. It is by no means complete and it is quite likely that you need to modify generated tasks.py before it is useful, with the exception of very trivial Makefiles. It should still save you from some manual work, and for me it was a good excuse to something bit more serious with Haskell than just to dabble with ghci solving Project Euler problems.

Install instructions

clone the repository, then do

stack build && \
  stack install && \
  echo "installed to $(stack path --local-bin)"

If you don't have Stack setup already, I recommend using ghcup to install it.

Sample run

With Makefile contents of

.PHONY: deps
TEST_DIR := tests

.PHONY: dev-init
dev-init: .check-poetry deps quick-test ## prepare project ready for development

.PHONY: .check-poetry
.check-poetry:
	@command -v poetry 2>/dev/null || (echo "poetry not found, please see https://python-poetry.org/docs/#installation"; exit 1)

.PHONY: deps
deps:  ## install dependencies
	poetry install --no-root

.PHONY: quick-test
quick-test: opts ?= --durations 3
quick-test:
	pytest $(opts) -m "not slow" $(TEST_DIR)

Resulting output:

from invoke import task

TEST_DIR = "tests"

@task
def deps(c):
    c.run('poetry install --no-root')


@task
def _check_poetry(c):
    c.run('command -v poetry 2>/dev/null || (echo "poetry not found, please see https://python-poetry.org/docs/#installation"; exit 1)')


@task
def quick_test(c):
    c.run(f'pytest {opts} -m "not slow" {TEST_DIR}')


@task(pre=[_check_poetry, deps, quick_test])
def dev_init(c):
    pass

Credits

Nicolas Mattias for his Makefile library, which I modified a bit to support parsing of top-level and inline comments

demake's Issues

Support Justfiles

Justifiles are nifty. We should support those too, esp. as they are really close to Makefile anyways

Homebrew recipe

Create Homebrew recipe so that users can easily install the system as well as get updates automatically

Support Shake files

Shake is even neater-er than other systems, tapping on all the goodies that Haskell sports while still being pretty lightweight, as seen in this example

at-chars (`@`) should be stripped from resulting python rules

I I have

foo:
	@bar

inv task should have only c.run("bar") without @ char

Automated release when merged to master

Create job/workflow which builds release for MacOS and Linux when build and tests are successful

Add in-target comments as docstrings in PyInvoke tasks

Meaning when I have

foo:   ## Execute foo
    run_cmd

it should be compiled to

@task
def foo(c):
    """Execute foo"""
    c.run("run_cmd")

Pass per-target options as invoke args

If we have a rule like

foo: opts ?= --fast -W
foo:
    build $(opts)

we could try parsing those opts and pass to invoke with some assumptions like this:

@task
def foo(c, fast: bool = True, W: bool = True):
    c.run("build")

Not sure if this is the way to go though, as the two formats are not isomorphic; using example above, Make would treat opts just as an unstructured string allowing use of positional args, which invoke doesn't have

build pipeline could definitely be faster

ensure packages are not re-downloaded/compiled every time a small change is made to single file