pybuilder / pybuilder

Software build automation tool for Python.

PyBuilder — an easy-to-use build automation tool for Python

PyBuilder is a software build tool written in 100% pure Python, mainly targeting Python applications.

PyBuilder is based on the concept of dependency based programming, but it also comes with a powerful plugin mechanism, allowing the construction of build life cycles similar to those known from other famous (Java) build tools.

PyBuilder is running on the following versions of Python 3.8, 3.9, 3.10, 3.11, 3.12 and PyPy 3.8 and 3.9.

See the GitHub Actions Workflow for version specific output.

Installing

PyBuilder is available using pip:

$ pip install pybuilder

For development builds use:

$ pip install --pre pybuilder

See the PyPI for more information.

Getting started

PyBuilder emphasizes simplicity. If you want to build a pure Python project and use the recommended directory layout, all you have to do is create a file build.py with the following content:

from pybuilder.core import use_plugin

use_plugin("python.core")
use_plugin("python.unittest")
use_plugin("python.coverage")
use_plugin("python.distutils")

default_task = "publish"

See the PyBuilder homepage for more details and a list of plugins.

Release Notes

The release notes can be found here. There will also be a git tag with each release. Please note that we do not currently promote tags to GitHub "releases".

Development

See Developing PyBuilder

pybuilder's Issues

Feature request : Compatibility with pip install from code drop

Hello awesome pybuilder team!

A nice feature of pip is that you can install directly from source, e.G.

pip install -e git+https://github.com/lakshmivyas/hyde.git#egg=hyde

This is becoming especially useful since GitHub decided to disable the upload functionality.
Unfortunately, pip is not very smart and will die with the above command if there is no setup.py file in the project, and AFAIK there is absolutely no way to override this behaviour.

Maybe you could expose functions that make it a breeze to create a dummy setup.py file for a pybuilder project, so that pip can be satisfied?
From my understanding, only setup.py egg_info and setup.py develop would be needed.

Regards,
max

add support for versions within install_dependencies

It would be nice if there is a way to specify the version for a dependency.
Something like:

project.depends_on('flask', '>=0.10.1')

[distutils-plugin] Should package modules also, not just packages

Currently, as @esc and @ollihoo found out, the distutils plugin will only package python sources if there are in a package, e.g.

--src
-------main
----------------python
-------------------------------foo
---------------------------------------__init__.py
---------------------------------------something.py

But if there are only modules, e.G.

--src
-------main
----------------python
-------------------------------foo.py
-------------------------------bar.py

then pybuilder will create an "empty" source distribution.
The reason is that modules must be added to setup.py with py_modules, something the distutils-plugin currently does not do at all.

Document execute_tool_on_source_files

This is an important API function for plugins because it forwards output in case of an error, based on $name_verbose_output.

Extend python.unittest plugin to support a unittest glob, or a unittest_file_prefix

First of all thanks for creating PyBuilder. I'm in the process of converting a project at work to use it and it's great!

Some tools we use expect unittests to have a specific file prefix, rather than a file suffix. What do you think about supporting an optional unittest_file_prefix property as well as unittest_file_suffix? Or perhaps a unittest_file_glob property? While we can work around this, it seems like others might be able to use this flexibility and it would help us integrate with some other tools we don't have source code for (PyCharm, for one.)

If your interested, I'd be willing to fork pybuilder and implement which ever semantics you prefer for your consideration. Obviously it would require a new variant of discover_modules, but that seems doable.

python.core plugin properties not documented (e.g. source directory override)

Unlike pybuilder.unittest plugin that has a documentation page about how src/unittest/python can be overridden via the project property dir_source_unittest_python, if I no longer want to use src/main/python as source directory, there is only a mention of this being possible in the tutorial.

I assume it's something like

    project.set_property("dir_source_main_python", "my/lovely/directory")

Need to pimp the integration test output when running in parallel

Currently integration tests running in parallel can be confusing, because there is no feedback when a test actually finishes. At least two different users thought that

Running integration test xyz

means that the test successfully completed, although it was just started.

Ideas :

• display start/finish for each test instead of just start
• dynamic progress bar with colors for running/finished/not started

PyCharm-PlugIn

A PyCharm plugin would really be nice. Right now I have to manually mark src/main/python and src/unittest/python as python source directories.

Allow loading of external plugins

As a pybuilder user I want to be able to ship a plugin that is independent from the pybuilder sources (e.G. install from PyPI) but can be used nonetheless.

Skeleton Creation using `pyb create`

To stop us from discussing this further in #40 I create a new issue here.

@zenweasel suggested to create a new task pyb create which would setup a basic project (create directories etc...).

Generate project files for PyCharm

PyCharm doesn't really understand where things are installed to when I open up a pybuilder project. I'm wondering whether a generated project file can help pycharm identify this information?

install_dependencies_plugin should be able to set default timeout

I want to set the default HTTP timeout for pip commands run by the install_dependencies plugin, e.G.

pip install foo --default-timeout 120 (which sets the HTTP timeout to 120 seconds).

with something like

project.set_property('install_dependencies_default_http_timeout', 120)

This is due to our cheeseshop being a bit slow due to network restrictions..

Link to documentation is broken

README links to http://pybuilder.github.com/documentation/ which gives 404 error

Read ignored files from gitignore

Pybuilder should read ignored files from .gitignore file.

Asciinema demos

As a potential PyBuilder user I would like to see examples running to quickly gain an overview of the benefits PyBuilder offers.

Sublime text plugin

As a PyBuilder user working with sublime text I want a ST3-compatible pybuilder plugin to be able to run pyb without leaving the editor.

The following should be configurable:

• path to python interpreter (use the python_interpreter setting from anaconda and sublimepythonide)
• task(s) to run, default tasks as a default

Output should go to a scratch view.

Revive Django Plugin!

What are we going to do with the python.django plugin?
It's not doing a lot right now ...and what it is doing is buggy.

need to document pyb --start-project

Support Mamba BDD tests

It'd be great to be able to run Mamba tests with PyBuilder.

Mamba is a BDD style testing library.

Command-line cop-out plugin

To increase adoption of pybuilder without supporting every possible thing Pythonistas would like to do under the sun, we'll want a python.exec_cmd plugin that lets users pretty much define their own arbitrary shell commands and incorporate them as a part of the build process.

Liken this to the Maven exec plugin

Make sure module pythonbuilder has been removed.

Module pythonbuilder should die with release 0.10.0

Document "developing for pybuilder"

flake8 PlugIn: prints and counts every error twice

Build failed after upgrade to 0.9.15

BUILD FAILED - No such property: 'dir_source_integrationtest_python'

This happened because my project committer has no integration tests.
Therefore the property is not set. The property is set in the integrationtest_plugin.

I guess if you would have a project without unittests you would experience the same since the property dir_source_unittest_python is set in the unittest_plugin.

Problem running on Windows

When trying to build on windows using PyBuilder I always get a 'this file is being used by another process' error. How comes?!

$ pyb run_unit_tests
←[1mPYBUILDER Version 0.9.12
←[0;0mBuild started at 2013-10-28 15:16:36
------------------------------------------------------------
←[1m[INFO] ←[0;0m Building Awesome Project version 0.0.1
←[1m[INFO] ←[0;0m Executing build in c:\Users\Hok Shun Poon\awesome_project
←[1m[INFO] ←[0;0m Going to execute task run_unit_tests
------------------------------------------------------------
←[1;31mBUILD FAILED - [Error 32] The process cannot access the file because it i
s being used by another process: 'c:\\users\\hokshu~1\\appdata\\local\\temp\\tmp
5ojiry'
←[0;0m------------------------------------------------------------
Build finished at 2013-10-28 15:16:36
Build took 0 seconds (241 ms)

Thanks!

Install Dependencies not running before unittests

I was just setting up a little lab with the sample projects included with package just to see how the package worked, etc.

Create a separate directory called pybuilder-samples, copied the sample files in there, made a virtualenv, pip install pybuilder into it and then ran them.

It looks like maybe these are set up with the some "fill in the blanks" stuff but I wanted to get these working before doing anything more complicated.

When I try to run "simple-python-project" I get an error saying it can't find mockito. I had adjusted the build.py to be the same as in the tutorial:

from pybuilder.core import init, use_plugin

use_plugin("python.core")
use_plugin("python.install_dependencies")
use_plugin("python.unittest")

default_task = "publish"

@init
def initialize(project):
    project.build_depends_on('mockito')

When I run pyb under a debugger I see that it loads all the modules including "install_dependencies" however, none of the modules that actually install dependencies actually get executed. "initialized_install_dependencies_plugin" and "check_pip_available" are run but none of the "installl_*" methods are ever executed.

I am not convinced this is a bug but something I have missed in configuration.

Here is the structure of my "samples" project:

|____build.py
|____build.pyc
|____src
| |____main
| | |____python
| | | |____helloworld.py
| | | |____helloworld.pyc
| |____unittest
| | |____python
| | | |____helloworld_tests.py
| | | |____helloworld_tests.pyc
|____target
| |____dist
| | |____simple-python-app-1.0-SNAPSHOT
| | | |____helloworld.py
| | | |____helloworld.pyc
| |____reports
| | |____unittest
| | |____unittest.json

Measure branch coverage

Ned Batchelder's coverage tool (which is the one we are using) allows to measure the branch coverage via a option ---branch. There should be a property to switch this on.

Unittest verbose mode

Currently pyb run_unit_tests just says that tests failed without any additional messages.
Is there a verbose mode?

Right now it's not possible to use a single quote in build.py's description.

Unfortunately when we write the setup.py we are using a string as template.
This template contains single quotes for the description string, which means if you are using a single quote in the description the distutils plugin is generating invalid python. distutils_plugin line 44

deal with unittest2 somehow

unittest2 is a backport of the new features added to the unittest testing framework in Python 2.7. It adds a few features like setUpClass.

I am unsure if it's worth the hassle myself (setUpClass makes it easy to write bad tests that depend on each other) but there might be some good uses of the new features.
I think it should be possible to either use unittest2 by default (if it's there) or to switch this behaviour on.

Generate description from README, README.md, README.txt

As a python developer I want to create a nice description for PyPI.
Nearly every GitHub project comes with a README.md (or something like that).
It would be nice to have a simple way to generate the PyPI description from the GitHub flavored markdown / text file.
Within build.py

description = description_from_text_file("README")

or

description = description_from_github_markdown_file("README.md")

Import error doesn't show stack trace

During a refactoring a stumbled on the following phenomenon: an import error in the unittests does not print a stack trace (even not when verbose output is switched on).

ERROR] Error importing unittests: cannot import name SomeThing
------------------------------------------------------------
BUILD FAILED - Unable to execute unit tests.
------------------------------------------------------------

Therefore it's hard to see where the import error happens.

Ability to execute just one task

Sometimes it's convenient to just execute one task. There should be a way to quickly identify and run a single task from the command line, regardless where in the lifecycle it is.

frosted PlugIn

Switch to pip python API?

On some Linux distributions (namely fedora) pip comes as pip-python instead of pip. see fedora

Mockito cannot be installed on python3.3

I am unable to install mockito on a fresh, clean python3.3 install. Travis also fails.
I think we should get rid of mockito - it seems to use distribute bootstrapping with an old version of distribute (distribute-0.6.10). The current distribute version is 0.7.3

@aelgru what do you think?

Task list broken on python3.3

Due to python3's more consistent ordering checks, tasks are not sortable anymore (and the task listing command tries to sort them).

mriehl@ixion:~$ virtualenv v3 -p `which python3.3`
Running virtualenv with interpreter /usr/bin/python3.3
Using base prefix '/usr'
New python executable in v3/bin/python3.3
Also creating executable in v3/bin/python
Installing distribute.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.
Installing pip................done.
mriehl@ixion:~$ . v3/bin/activate
(v3)mriehl@ixion:~$ pip install pybuilder
Downloading/unpacking pybuilder
  Downloading pybuilder-0.9.8.tar.gz
  Running setup.py egg_info for package pybuilder

Installing collected packages: pybuilder
  Running setup.py install for pybuilder
    changing mode of build/scripts-3.3/pyb from 664 to 775

    changing mode of /home/mriehl/v3/bin/pyb to 775
Successfully installed pybuilder
Cleaning up...
(v3)mriehl@ixion:~$ pyb -t
Tasks found for project "mriehl":
Traceback (most recent call last):
  File "/home/mriehl/v3/bin/pyb", line 23, in <module>
    sys.exit(pybuilder.cli.main(*sys.argv[1:]))
  File "/home/mriehl/v3/lib/python3.3/site-packages/pybuilder/cli.py", line 275, in main
    print_list_of_tasks(reactor)
  File "/home/mriehl/v3/lib/python3.3/site-packages/pybuilder/cli.py", line 247, in print_list_of_tasks
    for task in sorted(tasks):
TypeError: unorderable types: Task() < Task()

pytddmon plugin

pytddmon is a plugin that runs unit tests after every change and gives constant feedback about it with a simple widget.

Currently I use something like

PYTHONPATH=../../main/python/ ../../../venv/bin/python ~/workspace/pytddmon/src/pytddmon.py

in my unittest folder but that's a bit crazy. Pybuilder could encapsulate this for me and provide tasks like start_tdd_monitoring and stop_tdd_monitoring (nonblocking of course).

Currently pytddmon is not on the cheeseshop but when it becomes available I'll start work on this ASAP.

Document Plugins

Documented plugins:

• python.coverage
• python.distutils
• python.django
• python.flake8
• python.pep8
• python.install_dependencies
• python.pychecker
• python.Pydev
• python.pylint
• python.pymetrics
• python.unittest
• python.integrationtest
• copy_resources
• filter_resources
• source_distribution

Compatibility with nosetests

I want to be able to run nosetests as well.
How do I override "src/main/python" path?

Support PyTests

I prefer to write tests in PyTest format rather than nosetests because I like not having to abide by an API. Can pyb be made to detect and run these PyTests too?

make maven analogy more explicit

As note by @aelgru the abstract tasks provided by pybuilder (e.G. verify, analyze, publish, package, ...) are very similar to the maven lifecycle.

There should be more overall emphasis on this in the documentation.

Linting should run on all source files

Currently the flake8 plugin will only lint files in $dir_source_main_python which does not include the test sources (unittest, integrationtest).

This sucks, because readability in test sources is very much an issue!

The plugins use execute_tool_on_source_files from python_plugin_helper.
A quick search reveals that this is used in linting plugins only, so IMHO execute_tool_on_source_files should run on all important source files (but definitely not on the project directory, as it would include the virtualenv files).

Test for installed pybuilder

We should have a test that checks if pybuilder is installed locally. This can lead to strange test failures because the imports pull from the local installation, but the tests are from the source. I think I remember @aelgru did this in shtub or so.

Custom tasks should know whether project sources have changed at all

I'm writing a custom task and am finding it difficult to determine whether the project sources or build scripts had changed since the task last ran.

This would be useful in speeding up repeated builds because certain phases don't need to perform any building if the artefact it produces (in the target directory) is already up to date. This is in accordance with the property that each execution phase in pybuilder should be idempotent.

The flake8_plugin should require prepare to run first

See this issue.
analyze currently does not enforce that prepare must run first, but it is prepare that creates the reports directory.
Since the flake8 plugin writes a report, it should require prepare to run first.

Easy to reproduce with pyb clean followed by pyb analyze with the flake8 plugin active.

Workaround: distutils sdist fails on some filesystems (like VirtualBox shared filesystem)

This is not a PyBuilder bug, but I thought this work around might be useful to other PyBuilder users. There is a long standing distutils bug that prevents distutils from working in certain environments. Basically, distutils assumes that all mounted filesystems support hardlinks if os.link exists. This is not a valid assumption:

BuildFailedException: Error while executing setup command sdist
------------------------------------------------------------
BUILD FAILED - Error while executing setup command sdist
------------------------------------------------------------

Here is a gist of a workaround that I use: file-sdist_hack-py

If you are encountering "error: Operation not permitted" errors in VirtualBox on the shared filesystem, including the gist's code in build.py should workaround the issue.

It's admittedly a hack as it monkey patches the distutil_plugin using too much knowledge of the distutil_plugin's internals. Then the monkey patched code generates a setup.py file that monkey patches the os module, deleting it's reference to link. But it works...

Verbose mode default?

I find myself constantly passing the -v flag to get feedback about failed tests or errors found during analysis.

I think that either the project should be verbose by default or the plugins should always give information about failures (when they lead to a BuildFailed at least, since we know that's the last thing that happens.).

The latter is probably better because it's not too intrusive.

I understand the need to focus on the big picture but when there is one error preventing me from building I would like to know what it is.

What do you think @aelgru ?

Self-Bootstrapping

It would be great if the build.py could be used to bootstrap pybuilder.

e.G.
./build.py should give me feedback that I need to install pybuilder if it's not installed, and run pybuilder if it is.

This would avoid discouraging people used to setup.py or manage.py since currently trying to run the build description does nothing useful or understandable.

Use ordered dictionaries.

Some of the integration tests require ordered dictionaries.

https://travis-ci.org/pybuilder/pybuilder/jobs/7599813