paylogic / py2deb

Python to Debian package converter

py2deb: Python to Debian package converter

The Python package py2deb converts Python source distributions to Debian binary packages (the ones used for installation). It uses pip-accel (based on pip) to download, unpack and compile Python packages. Because of this py2deb is compatible with the command line interface of the pip install command. For example you can specify packages to convert as command line arguments but you can also use requirement files if you want.

During the conversion process dependencies are automatically taken into account and converted as well so you don't actually have to use requirement files including transitive dependencies. In fact you might prefer not explicitly listing your transitive dependencies in requirement files because py2deb will translate the version constraints of Python packages into Debian package relationships.

The py2deb package is currently tested on CPython 2.7, 3.5, 3.6, 3.7 and PyPy 2 and 3. Unfortunately Python 3.8+ is not yet supported (see below). For usage instructions please refer to the documentation hosted on Read The Docs.

• Installation
• Usage
  • Command line
• Future improvements
  • Python 3.8+ compatibility
  • Installation of system wide files
  • Conversion of binary wheels
  • Full PEP-440 compatibility
• Similar projects
• Contact
• License

Installation

The py2deb package is available on PyPI, so installation is very simple:

$ pip install py2deb

There are some system dependencies which you have to install as well:

$ sudo apt-get install dpkg-dev fakeroot

Optionally you can also install Lintian (which is not a hard dependency but more of a "nice to have"):

$ sudo apt-get install lintian

When Lintian is installed it will be run automatically to sanity check converted packages. This slows down the conversion process somewhat but can be very useful, especially when working on py2deb itself. Currently py2deb doesn't fail when Lintian reports errors, this is due to the unorthodox ways in which py2deb can be used. This may change in the future as py2deb becomes more mature.

Usage

There are two ways to use the py2deb package: As the command line program py2deb and as a Python API. For details about the Python API please refer to the API documentation hosted on Read the Docs. The command line interface is described below.

Command line

Usage: py2deb [OPTIONS] ...

Convert Python packages to Debian packages according to the given command line options (see below). The command line arguments are the same as accepted by the "pip install" command because py2deb invokes pip during the conversion process. This means you can name the package(s) to convert on the command line but you can also use "requirement files" if you prefer.

If you want to pass command line options to pip (e.g. because you want to use a custom index URL or a requirements file) then you will need to tell py2deb where the options for py2deb stop and the options for pip begin. In such cases you can use the following syntax:

$ py2deb -r /tmp -- -r requirements.txt

So the "--" marker separates the py2deb options from the pip options.

Supported options:

Option	Description
-c, --config=FILENAME	Load a configuration file. Because the command line arguments are processed in the given order, you have the choice and responsibility to decide if command line options override configuration file options or vice versa. Refer to the documentation for details on the configuration file format. The default configuration files /etc/py2deb.ini and ~/.py2deb.ini are automatically loaded if they exist. This happens before environment variables and command line options are processed. Can also be set using the environment variable $PY2DEB_CONFIG.
-r, --repository=DIRECTORY	Change the directory where *.deb archives are stored. Defaults to the system wide temporary directory (which is usually /tmp). If this directory doesn't exist py2deb refuses to run. Can also be set using the environment variable $PY2DEB_REPOSITORY.
--use-system-package=PYTHON_PACKAGE_NAME,DEBIAN_PACKAGE_NAME	Exclude a Python package (the name before the comma) from conversion and replace references to the Python package with a specific Debian package name. This allows you to use system packages for specific Python requirements.
--name-prefix=PREFIX	Set the name prefix used during the name conversion from Python to Debian packages. Defaults to "python". The name prefix and package names are always delimited by a dash. Can also be set using the environment variable $PY2DEB_NAME_PREFIX.
--no-name-prefix=PYTHON_PACKAGE_NAME	Exclude a Python package from having the name prefix applied during the package name conversion. This is useful to avoid awkward repetitions.
--rename=PYTHON_PACKAGE_NAME,DEBIAN_PACKAGE_NAME	Override the package name conversion algorithm for the given pair of package names. Useful if you don't agree with the algorithm :-)
--install-prefix=DIRECTORY	Override the default system wide installation prefix. By setting this to anything other than "/usr" or "/usr/local" you change the way py2deb works. It will build packages with a file system layout similar to a Python virtual environment, except there will not be a Python executable: The packages are meant to be loaded by modifying Python's module search path. Refer to the documentation for details. Can also be set using the environment variable $PY2DEB_INSTALL_PREFIX.
--install-alternative=LINK,PATH	Use Debian's "update-alternatives" system to add an executable that's installed in a custom installation prefix (see above) to the system wide executable search path. Refer to the documentation for details.
--python-callback=EXPRESSION	Set a Python callback to be called during the conversion process. Refer to the documentation for details about the use of this feature and the syntax of EXPRESSION. Can also be set using the environment variable $PY2DEB_CALLBACK.
--report-dependencies=FILENAME	Add the Debian relationships needed to depend on the converted package(s) to the given control file. If the control file already contains relationships the additional relationships will be added to the control file; they won't overwrite existing relationships.
-y, --yes	Instruct pip-accel to automatically install build time dependencies where possible. Refer to the pip-accel documentation for details. Can also be set using the environment variable $PY2DEB_AUTO_INSTALL.
-v, --verbose	Make more noise :-).
-h, --help	Show this message and exit.

Future improvements

The following sections list possible improvements to the project:

• Python 3.8+ compatibility
• Installation of system wide files
• Conversion of binary wheels
• Full PEP-440 compatibility

Python 3.8+ compatibility

The py2deb project builds on top of pip-accel, which was developed between 2013 and 2015 on top of pip >= 7.0, < 7.2. Since that time pip has grown enormously: At the time of writing (in August 2020) we're now at pip 20!

None of the improvements made between pip 7-20 are available in pip-accel and py2deb and this has become somewhat of a glaring issue that plenty of users have run into (see #17, #18, #27 and #31).

Known issues being caused by this include:

• The old pip version prevents Python 3.8+ compatibility.
• The old pip version doesn't know about python_requires metadata provided by PyPI and this forces users to maintain constraints files themselves, even though this shouldn't be necessary.
• While pip-accel supports installation from wheels, it was never exposed via the Python API and so py2deb lacks support for converting wheels (it currently needs source distributions).

The current state of affairs is best summarized in this comment. I'm hoping to complete the upgrade to newer pip and pip-accel releases in the coming weeks (as of this writing in August 2020) but can't commit to a date.

Installation of system wide files

Find a way to facilitate (explicit / opt-in) installation of system wide files (not related to Python per se) based on a Python distribution? This could significantly reduce the need for "wrapper packages" that basically just pull in packages converted by py2deb and drop a few configuration files into place.

Related issues:	See issue #7 for a related discussion.

Conversion of binary wheels

Investigate the feasability of supporting conversion of binary wheels. Slowly but surely the Python community seems to be gravitating towards (binary) wheels and once gravity has shifted we don't want to be left in the dust! 😉

Full PEP-440 compatibility

Dive into PEP-440 and see if it can be fully supported? Then this question on Reddit can finally get a satisfying answer 🙂.

Similar projects

There are several projects out there that share similarities with py2deb, for example I know of stdeb, dh-virtualenv and fpm. The documentation includes a fairly detailed comparison with each of these projects.

Contact

The latest version of py2deb is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For questions, bug reports, suggestions, etc. please create an issue on GitHub.

License

This software is licensed under the MIT license.

© 2020 Peter Odding, Arjan Verwer and Paylogic International.

py2deb's Issues

Fields override from default stdeb.cfg section

Hello,

py2deb/py2deb/package.py

Line 607 in 05497a5

if parser.has_section(section_name):

Just playing with manual fields override and noticed it didn't work with the DEFAULT section but was fine with section matching the python package name. Maybe has to do with the fact that has_section() don't work for the DEFAULT section ?

https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.has_section

RequirementParseError seen running CLI

Trying to convert newer pip package into debian for older Ubuntu. Seeing this error:

2020-09-10 17:21:22 laas-greggmi pip_accel[2201] INFO Unpacking distribution(s) ..
2020-09-10 17:21:22 laas-greggmi pip_accel[2201] INFO Executing command: pip install --download=/users/greggmi/.pip-accel/sources --find-links=file:///users/greggmi/.pip-accel/sources --no-binary=:all: --no-index --no-clean --build-directory=/tmp/pip-accel-build-dir-m1T4oR --ignore-installed pyvmomi==6.0.0.2016.6
2020-09-10 17:21:22 laas-greggmi pip.commands.install[2201] INFO Ignoring indexes: https://pypi.python.org/simple
2020-09-10 17:21:22 laas-greggmi pip.req.req_set[2201] INFO Collecting pyvmomi==6.0.0.2016.6
2020-09-10 17:21:22 laas-greggmi pip.download[2201] INFO File was already downloaded /users/greggmi/.pip-accel/sources/pyvmomi-6.0.0.2016.6.tar.gz
2020-09-10 17:21:22 laas-greggmi pip.req.req_set[2201] INFO Collecting requests>=2.3.0 (from pyvmomi==6.0.0.2016.6)
2020-09-10 17:21:22 laas-greggmi pip.download[2201] INFO File was already downloaded /users/greggmi/.pip-accel/sources/requests-2.24.0.tar.gz
2020-09-10 17:21:23 laas-greggmi pip.req.req_set[2201] INFO Collecting six>=1.7.3 (from pyvmomi==6.0.0.2016.6)
2020-09-10 17:21:23 laas-greggmi pip.download[2201] INFO File was already downloaded /users/greggmi/.pip-accel/sources/six-1.15.0.tar.gz
2020-09-10 17:21:23 laas-greggmi pip.req.req_set[2201] INFO Collecting chardet<4,>=3.0.2 (from requests>=2.3.0->pyvmomi==6.0.0.2016.6)
2020-09-10 17:21:23 laas-greggmi pip.download[2201] INFO File was already downloaded /users/greggmi/.pip-accel/sources/chardet-3.0.4.tar.gz
2020-09-10 17:21:24 laas-greggmi pip.req.req_set[2201] INFO Collecting idna<3,>=2.5 (from requests>=2.3.0->pyvmomi==6.0.0.2016.6)
2020-09-10 17:21:24 laas-greggmi pip.download[2201] INFO File was already downloaded /users/greggmi/.pip-accel/sources/idna-2.10.tar.gz
2020-09-10 17:21:24 laas-greggmi pip.req.req_set[2201] INFO Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests>=2.3.0->pyvmomi==6.0.0.2016.6)
2020-09-10 17:21:24 laas-greggmi pip.download[2201] INFO File was already downloaded /users/greggmi/.pip-accel/sources/urllib3-1.25.10.tar.gz
2020-09-10 17:21:24 laas-greggmi py2deb.cli[2201] ERROR Caught an unhandled exception!
Traceback (most recent call last):
  File "/users/greggmi/.local/lib/python2.7/site-packages/py2deb/cli.py", line 200, in main
    archives, relationships = converter.convert(arguments)
  File "/users/greggmi/.local/lib/python2.7/site-packages/py2deb/converter.py", line 736, in convert
    self.packages_to_convert = list(self.get_source_distributions(pip_install_arguments))
  File "/users/greggmi/.local/lib/python2.7/site-packages/py2deb/converter.py", line 793, in get_source_distributions
    for requirement in self.pip_accel.get_requirements(arguments):
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip_accel/__init__.py", line 297, in get_requirements
    return self.unpack_source_dists(arguments, use_wheels=use_wheels)
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip_accel/__init__.py", line 393, in unpack_source_dists
    requirements = self.get_pip_requirement_set(arguments, use_remote_index=False, use_wheels=use_wheels)
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip_accel/__init__.py", line 497, in get_pip_requirement_set
    requirement_set = command.run(opts, args)
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/commands/install.py", line 294, in run
    requirement_set.prepare_files(finder)
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/req/req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/req/req_set.py", line 561, in _prepare_file
    set(req_to_install.extras) - set(dist.extras)
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2774, in extras
    return [dep for dep in self._dep_map if dep]
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2591, in _dep_map
    dm.setdefault(extra,[]).extend(parse_requirements(reqs))
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2924, in parse_requirements
    "version spec")
  File "/users/greggmi/.local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2889, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])
RequirementParseError: Expected version spec in ipaddress; python_version=='2.7' at ; python_version=='2.7'

I couldn't find any much googling this, but I think this is the culprit: https://github.com/urllib3/urllib3/blob/3c3fb0299f5e56613003bc293a9a9082e264c982/dev-requirements.txt

debian package of numpy for debian wheezy with py2deb

Hi, thanks for py2deb, it's exactly what I need now.

I'm trying to create a debian package of numpy for debian wheezy for the version v1.14.1 (the official debian package is an older version). However, I'm having these errors:

# py2deb -r /output_debs/ numpy
2018-03-09 16:44:28 4222eefebe82 pip_accel[21987] INFO Unpacking distribution(s) ..
2018-03-09 16:44:28 4222eefebe82 pip_accel[21987] INFO Executing command: pip install --download=/var/cache/pip-accel/sources --find-links=file:///var/cache/pip-accel/sources --no-binary=:all: --no-index --no-clean --build-directory=/tmp/pip-accel-build-dir-P1smGj --ignore-installed numpy
2018-03-09 16:44:28 4222eefebe82 pip.commands.install[21987] INFO Ignoring indexes: https://pypi.python.org/simple
2018-03-09 16:44:29 4222eefebe82 pip.req.req_set[21987] INFO Collecting numpy
2018-03-09 16:44:29 4222eefebe82 pip.download[21987] INFO File was already downloaded /var/cache/pip-accel/sources/numpy-1.14.1.zip
2018-03-09 16:44:29 4222eefebe82 pip.commands.install[21987] INFO Successfully downloaded numpy
2018-03-09 16:44:29 4222eefebe82 pip_accel[21987] INFO Finished unpacking 1 distribution in 0.84 seconds.
2018-03-09 16:44:29 4222eefebe82 pip_accel.bdist[21987] INFO Building numpy (1.14.1) binary distribution ..
2018-03-09 16:44:30 4222eefebe82 pip_accel.bdist[21987] WARNING Build of numpy (1.14.1) failed, falling back to alternative method ..
2018-03-09 16:44:30 4222eefebe82 pip_accel.bdist[21987] INFO Building numpy (1.14.1) binary distribution ..
2018-03-09 16:44:30 4222eefebe82 pip_accel.bdist[21987] WARNING Build of numpy (1.14.1) failed, checking for missing dependencies ..
2018-03-09 16:44:30 4222eefebe82 pip_accel.deps[21987] INFO Checking for known dependencies of numpy ..
2018-03-09 16:44:30 4222eefebe82 pip_accel.deps[21987] INFO No known dependencies... Maybe you have a suggestion?
2018-03-09 16:44:30 4222eefebe82 py2deb.cli[21987] ERROR Caught an unhandled exception!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/py2deb/cli.py", line 200, in main
    archives, relationships = converter.convert(arguments)
  File "/usr/local/lib/python2.7/dist-packages/py2deb/converter.py", line 546, in convert
    archive = package.convert()
  File "/usr/local/lib/python2.7/dist-packages/py2deb/package.py", line 376, in convert
    virtualenv_compatible=False,
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/bdist.py", line 437, in install_binary_dist
    for member, from_handle in members:
  File "/usr/local/lib/python2.7/dist-packages/py2deb/package.py", line 503, in transform_binary_dist
    for member, handle in self.converter.pip_accel.bdists.get_binary_dist(self.requirement):
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/bdist.py", line 87, in get_binary_dist
    raw_file = self.build_binary_dist(requirement)
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/bdist.py", line 217, in build_binary_dist
    return self.build_binary_dist_helper(requirement, ['bdist', '--formats=gztar'])
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/bdist.py", line 282, in build_binary_dist_helper
    name=requirement.name, version=requirement.version)
BuildFailed: Failed to build numpy (1.14.1) binary distribution!

Please check the build output because it will probably provide a hint about what went wrong.

Build output:

Running from numpy source directory.
`setup.py bdist` is not supported
Add `--force` to your command to use it anyway if you must (unsupported).

Any advice?
Thanks.

py2deb no longer seems to handle python/debian versions

So for a long time my python project had a setup.py with specific version of python packages.

install_requires=[
    'flask==0.10.1',  # or whatever
    ...

Everything has worked fine for 6ish months and now suddenly I'm getting these errors:

2016-02-12 01:26:22 vagrant-ubuntu-trusty-64 py2deb.cli[6284] ERROR Caught an unhandled exception!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/py2deb/cli.py", line 192, in main
    archives, relationships = converter.convert(arguments)
  File "/usr/local/lib/python2.7/dist-packages/py2deb/converter.py", line 524, in convert
    archive = package.convert()
  File "/usr/local/lib/python2.7/dist-packages/py2deb/package.py", line 383, in convert
    dependencies = [python_version()] + self.debian_dependencies
  File "/usr/local/lib/python2.7/dist-packages/cached_property.py", line 26, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python2.7/dist-packages/py2deb/package.py", line 295, in debian_dependencies
    version = self.converter.transform_version(self, requirement.project_name, version)
  File "/usr/local/lib/python2.7/dist-packages/py2deb/converter.py", line 656, in transform_version
    """, name=normalized_name, count=num_matches, matches=matching_packages))
Exception: Expected requirement set to contain exactly one Python package whose name can be normalized to flask but encountered 0 packages instead! (matching packages: [])

DeprecationWarning for imp in py2deb hooks file

/var/lib/dpkg/info/{package}.postinst:39: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

Appears during package installation for all maintainer scripts (prerm, postinst, etc.)

Problem with PEP 440 pre-release versions

The versioning scheme defined in PEP 440 (https://www.python.org/dev/peps/pep-0440/) includes pre-release versions. For example, version 1.0a2 is the second alpha version of the 1.0 release. Thus, version 1.0a2 comes before version 1.0 (1.0a2 < 1.0).
However, the Debian versioning works differently (https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version). Here, it holds that 1.0 < 1.0a2.

py2deb doesn't handle this inconsistency in any way, which leads to problems. For example, let's take the PyPI package pyramid 1.7 (https://pypi.python.org/pypi/pyramid/1.7), which requires venusian (>=1.0a3). There is already a version 1.0 of venusian. Thus, when we invoke py2deb on pyramid 1.7, it creates (among others) a .deb for pyramid 1.7 and one for venusian 1.0. The pyramid .deb specifies a dependency on venusian (>= 1.0a3). When we now try to install those .debs with dpkg, this dependency is checked. dpkg looks for a venusian .deb with version >= 1.0a3, but finds only one with version 1.0, which (according to Debian versioning) is less than 1.0a3. Thus the install fails.

I think the quickest fix would be to replace to simply insert a ~ before every pre-release part (e.g. 1.0a2 becomes 1.0~a2). According to Debians versioning, ~ is smaller than the empty string, so this would fix the problem. There might be other problematic inconsistencies between PEP 440 and Debian versioning, though.

py2deb does not seem to work with the latest coloredlogs

Reverting to coloredlogs v4.0 makes it work fine. Since the requirements.txt of py2deb specifies >=0.5, coloredlogs v5.0 is installed. Which does not work, because of:

File "xxxx/py2deb/cli.py", line 215, in usage
    lines[i] = coloredlogs.ansi_text(line, color='green')
AttributeError: 'module' object has no attribute 'ansi_text'

Steps to reproduce: run py2deb without arguments. Running py2deb --help hides the traceback, but still fails. Running a valid command does not seem to break py2deb, because the help text printing logic does not seem to be called.

Error is present on both Python 2.7.11 and Python 3.5.1. Both installs were done with pip install py2deb in a clean virtualenv.

platform.linux_distribution not exists

the error:

Traceback (most recent call last):
File "/home/fredlins/.local/lib/python3.8/site-packages/py2deb/cli.py", line 200, in main
archives, relationships = converter.convert(arguments)
File "/home/fredlins/.local/lib/python3.8/site-packages/py2deb/converter.py", line 735, in convert
self.packages_to_convert = list(self.get_source_distributions(pip_install_arguments))
File "/home/fredlins/.local/lib/python3.8/site-packages/py2deb/converter.py", line 792, in get_source_distributions
for requirement in self.pip_accel.get_requirements(arguments):
File "/home/fredlins/.local/lib/python3.8/site-packages/pip_accel/init.py", line 297, in get_requirements
return self.unpack_source_dists(arguments, use_wheels=use_wheels)
File "/home/fredlins/.local/lib/python3.8/site-packages/pip_accel/init.py", line 393, in unpack_source_dists
requirements = self.get_pip_requirement_set(arguments, use_remote_index=False, use_wheels=use_wheels)
File "/home/fredlins/.local/lib/python3.8/site-packages/pip_accel/init.py", line 497, in get_pip_requirement_set
requirement_set = command.run(opts, args)
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/commands/install.py", line 245, in run
with self._build_session(options) as session:
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/basecommand.py", line 65, in _build_session
session = PipSession(
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/download.py", line 316, in init
self.headers["User-Agent"] = user_agent()
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/download.py", line 92, in user_agent
zip(["name", "version", "id"], platform.linux_distribution()),
AttributeError: module 'platform' has no attribute 'linux_distribution'

I dont know why you call this method linux_distribution, because not exists.

Builds on Travis CI not passing

While developing py2deb I've been extending and running the test suite continuously for months now and that was working great. When I published py2deb on GitHub a couple of days ago I got my first chance to run the py2deb test suite on Travis CI and I found out and that it fails consistently on all Python versions :-(.

Because there are multiple ways to fix this problem and all of them are somewhat involved / complex, I will use this GitHub issue to summarize my thoughts and weigh the pros and cons of each approach.

My local testing environment

To summarize my local testing environment:

• Ubuntu Linux 12.04, 64 bits
• Default Python version is Python 2.7.3 installed from official repositories.
• Additional Python versions installed from deadsnakes PPA:
  • Python 2.6.9
  • Python 3.4.2
• Using Tox to run test suite against Python 2.6, 2.7 and 3.4

The Travis CI testing environment

To summarize what I know and have inferred (assumed) about the Travis CI testing environment:

• Travis CI workers are also running Ubuntu 12.04
• I believe that in the past Travis CI was using the deadsnakes PPA
  • I can't find anything about this in the online documentation (just dead references from external sites)
  • My findings with the py2deb test suite seem to point out that Travis CI is using custom compiled Python binaries, even for Python 2.7 (the system default on Ubuntu 12.04)

At present it seems like Travis CI is using a Chef cookbook that uses pyenv to install custom compiled Python binaries to run test suites against. Some references:

• https://github.com/travis-ci/travis-cookbooks/blob/master/vm_templates/common/python.yml references python::pyenv
• https://github.com/travis-ci/travis-cookbooks/blob/master/ci_environment/python/recipes/pyenv.rb shows what python::pyenv does
• All of this eventually leads to https://github.com/yyuu/pyenv which gives more background information about pyenv in general

Why the py2deb test suite fails

Due to its goals py2deb obviously integrates quite a bit with Debian Python packaging. One such bit of integration is my (ab)use of dpkg-shlibdeps. It seems to me that the main issue on Travis CI right now is that dpkg-shlibdeps immediately errors out because *.so files are compiled against a Python binary that is not packaged. The same basic issue is described in the StackOverflow thread dpkg-shlibdeps: error: no dependency information found for.

Ways to get the test suite running

1. Don't use the Travis CI installed Python versions, instead start from a clean Ubuntu 12.04 install, manually use the deadsnakes PPA to get Python 2.6 and 3.4 installed and use Tox to run the test suite against all of them.
   • Pro: Fairly quickly to get going again, allowing me to focus on "more important things".
   • Con: When done the naive way this would force the 2.6/2.7/3.4 tests to run in serial, but that can be fixed by creative use of a a build matrix based on custom environment variables.
   • Con: Not using Travis CI the way it's meant to be used. Then again, Travis CI is just a tool, and their defaults don't necessarily hold for every project in the world (which they acknowledge).
2. Instead of side stepping Travis CI's use of custom compiled Python binaries, dive deeper into dpkg-shlibdeps and see if we can get it to ignore these issues but keep looking for other dependencies.
   • Pro: I looked into the Perl sources a bit and that seemed impossible, but the previously linked StackOverflow thread suggests using the --ignore-missing-info option. Maybe I'll use that as my first option (easiest to try out).
   • Con: Users of py2deb should realize when proper dependency information is not generated. In other words, when dpkg-shlibdeps fails I want py2deb to fail, because blindly marching on could mean a binary Debian package is generated which is missing essential dependencies on system libraries! So just ignoring error codes could get awkward.
3. The feature provided by the integration of dpkg-shlibdeps is very useful and I don't want to lose that, but clearly the way I'm running dpkg-shlibdeps is a huge hack waiting to bite me in the ass. I've been itching to replace dpkg-shlibdeps because of this, but as I mentioned I looked into its sources and there are literally thousands of lines of (for me) indecipherable Perl code underneath.
   • Pro: A reimplementation dedicated to py2deb could provide exactly what py2deb means, without the gross hacks required to get dpkg-shlibdeps running.
   • Con: Can I really be excused for wasting time porting all of that to Python, making the same bugs that inevitably occurred during the development of dpkg-shlibdeps? (a rhetorical question; I know the answer ;-) I'll also need to maintain all of this in later Debian/Ubuntu releases...

Right now I will investigate option two because it's the most light weight approach with the least potential to bite me (the py2deb project) in the ass later on.

Comparison to fpm

How does py2deb compare to fpm?

module 'platform' has no attribute 'linux_distribution'

py2deb -v crazydiskmark Tue 01 Sep 2020 07:27:41 PM UTC
2020-09-01 16:27:48 shadowBox pip_accel[89399] INFO Unpacking distribution(s) ..
2020-09-01 16:27:48 shadowBox pip_accel[89399] INFO Executing command: pip install --download=/home/fredlins/.pip-accel/sources --find-links=file:///home/fredlins/.pip-accel/sources --no-binary=:all: --no-index --no-clean --build-directory=/tmp/pip-accel-build-dir-nl_2re2q --ignore-installed crazydiskmark
2020-09-01 16:27:48 shadowBox pip.commands.install[89399] INFO Ignoring indexes: https://pypi.python.org/simple
2020-09-01 16:27:48 shadowBox py2deb.cli[89399] ERROR Caught an unhandled exception!
Traceback (most recent call last):
File "/home/fredlins/.local/lib/python3.8/site-packages/py2deb/cli.py", line 200, in main
archives, relationships = converter.convert(arguments)
File "/home/fredlins/.local/lib/python3.8/site-packages/py2deb/converter.py", line 736, in convert
self.packages_to_convert = list(self.get_source_distributions(pip_install_arguments))
File "/home/fredlins/.local/lib/python3.8/site-packages/py2deb/converter.py", line 793, in get_source_distributions
for requirement in self.pip_accel.get_requirements(arguments):
File "/home/fredlins/.local/lib/python3.8/site-packages/pip_accel/init.py", line 297, in get_requirements
return self.unpack_source_dists(arguments, use_wheels=use_wheels)
File "/home/fredlins/.local/lib/python3.8/site-packages/pip_accel/init.py", line 393, in unpack_source_dists
requirements = self.get_pip_requirement_set(arguments, use_remote_index=False, use_wheels=use_wheels)
File "/home/fredlins/.local/lib/python3.8/site-packages/pip_accel/init.py", line 497, in get_pip_requirement_set
requirement_set = command.run(opts, args)
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/commands/install.py", line 245, in run
with self._build_session(options) as session:
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/basecommand.py", line 65, in _build_session
session = PipSession(
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/download.py", line 316, in init
self.headers["User-Agent"] = user_agent()
File "/home/fredlins/.local/lib/python3.8/site-packages/pip/download.py", line 92, in user_agent
zip(["name", "version", "id"], platform.linux_distribution()),
AttributeError: module 'platform' has no attribute 'linux_distribution'

Malfunction on Python3.11 (Debian 12)

Installation

vitex@exiv:~/Projects/Packaging/Python$ python3.11 -m venv .venv
vitex@exiv:~/Projects/Packaging/Python$ source .venv/bin/activate
(.venv) vitex@exiv:~/Projects/Packaging/Python$ pip install py2deb
Collecting py2deb
  Using cached py2deb-5.0-py2.py3-none-any.whl (54 kB)
Collecting coloredlogs>=0.5
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Collecting deb-pkg-tools>=5.2
  Using cached deb_pkg_tools-8.4-py2.py3-none-any.whl (87 kB)
Collecting executor>=21.0
  Using cached executor-23.2-py2.py3-none-any.whl (85 kB)
Collecting humanfriendly>=8.0
  Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting pip-accel<=0.43,>=0.25
  Using cached pip_accel-0.43-py3-none-any.whl
Collecting pkginfo>=1.1
  Using cached pkginfo-1.9.6-py3-none-any.whl (30 kB)
Collecting property-manager>=2.3.1
  Using cached property_manager-3.0-py2.py3-none-any.whl (23 kB)
Collecting six>=1.6.1
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting fasteners>=0.14.1
  Using cached fasteners-0.18-py3-none-any.whl (18 kB)
Collecting cached-property>=0.1.5
  Using cached cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Collecting pip<7.2,>=7.0
  Using cached pip-7.1.2-py2.py3-none-any.whl (1.1 MB)
Requirement already satisfied: setuptools>=7.0 in ./.venv/lib/python3.11/site-packages (from pip-accel<=0.43,>=0.25->py2deb) (66.1.1)
Collecting verboselogs>=1.1
  Using cached verboselogs-1.7-py2.py3-none-any.whl (11 kB)
Installing collected packages: verboselogs, pip, cached-property, six, pkginfo, humanfriendly, fasteners, property-manager, coloredlogs, pip-accel, executor, deb-pkg-tools, py2deb
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed cached-property-1.5.2 coloredlogs-15.0.1 deb-pkg-tools-8.4 executor-23.2 fasteners-0.18 humanfriendly-10.0 pip-7.1.2 pip-accel-0.43 pkginfo-1.9.6 property-manager-3.0 py2deb-5.0 six-1.16.0 verboselogs-1.7

Problem

(.venv) vitex@exiv:~/Projects/Packaging/Python$ py2deb 
Traceback (most recent call last):
  File "/home/vitex/Projects/Packaging/Python/.venv/bin/py2deb", line 5, in <module>
    from py2deb.cli import main
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/py2deb/cli.py", line 137, in <module>
    from py2deb.converter import PackageConverter
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/py2deb/converter.py", line 35, in <module>
    from pip_accel import PipAccelerator
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip_accel/__init__.py", line 54, in <module>
    from pip_accel.bdist import BinaryDistributionManager
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip_accel/bdist.py", line 35, in <module>
    from pip_accel.caches import CacheManager
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip_accel/caches/__init__.py", line 25, in <module>
    from pip_accel.exceptions import CacheBackendDisabledError
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip_accel/exceptions.py", line 36, in <module>
    from pip_accel.utils import compact
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip_accel/utils.py", line 28, in <module>
    from pip.commands.uninstall import UninstallCommand
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip/__init__.py", line 13, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip/utils/__init__.py", line 23, in <module>
    from pip._vendor import pkg_resources
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__init__.py", line 32, in <module>
    import symbol
ModuleNotFoundError: No module named 'symbol'

Even update of pip and setuptools won't help :(

vitex@exiv:~/Projects/Packaging/Python$ python3.11 -m venv .venv
vitex@exiv:~/Projects/Packaging/Python$ source .venv/bin/activate
(.venv) vitex@exiv:~/Projects/Packaging/Python$ pip install --upgrade pip setuptools
Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.0.1)
Collecting pip
  Using cached pip-23.2.1-py3-none-any.whl (2.1 MB)
Requirement already satisfied: setuptools in ./.venv/lib/python3.11/site-packages (66.1.1)
Collecting setuptools
  Using cached setuptools-68.1.2-py3-none-any.whl (805 kB)
Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 66.1.1
    Uninstalling setuptools-66.1.1:
      Successfully uninstalled setuptools-66.1.1
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-23.2.1 setuptools-68.1.2


(.venv) vitex@exiv:~/Projects/Packaging/Python$ pip install --upgrade py2deb
Collecting py2deb
  Using cached py2deb-5.0-py2.py3-none-any.whl (54 kB)
Collecting coloredlogs>=0.5 (from py2deb)
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Collecting deb-pkg-tools>=5.2 (from py2deb)
  Using cached deb_pkg_tools-8.4-py2.py3-none-any.whl (87 kB)
Collecting executor>=21.0 (from py2deb)
  Using cached executor-23.2-py2.py3-none-any.whl (85 kB)
Collecting humanfriendly>=8.0 (from py2deb)
  Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting pip-accel<=0.43,>=0.25 (from py2deb)
  Using cached pip_accel-0.43-py3-none-any.whl
Collecting pkginfo>=1.1 (from py2deb)
  Using cached pkginfo-1.9.6-py3-none-any.whl (30 kB)
Collecting property-manager>=2.3.1 (from py2deb)
  Using cached property_manager-3.0-py2.py3-none-any.whl (23 kB)
Collecting six>=1.6.1 (from py2deb)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting fasteners>=0.14.1 (from executor>=21.0->py2deb)
  Using cached fasteners-0.18-py3-none-any.whl (18 kB)
Collecting cached-property>=0.1.5 (from pip-accel<=0.43,>=0.25->py2deb)
  Using cached cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Collecting pip<7.2,>=7.0 (from pip-accel<=0.43,>=0.25->py2deb)
  Using cached pip-7.1.2-py2.py3-none-any.whl (1.1 MB)
Requirement already satisfied: setuptools>=7.0 in ./.venv/lib/python3.11/site-packages (from pip-accel<=0.43,>=0.25->py2deb) (68.1.2)
Collecting verboselogs>=1.1 (from property-manager>=2.3.1->py2deb)
  Using cached verboselogs-1.7-py2.py3-none-any.whl (11 kB)
Installing collected packages: verboselogs, pip, cached-property, six, pkginfo, humanfriendly, fasteners, property-manager, coloredlogs, pip-accel, executor, deb-pkg-tools, py2deb
  Attempting uninstall: pip
    Found existing installation: pip 23.2.1
    Uninstalling pip-23.2.1:
      Successfully uninstalled pip-23.2.1
Successfully installed cached-property-1.5.2 coloredlogs-15.0.1 deb-pkg-tools-8.4 executor-23.2 fasteners-0.18 humanfriendly-10.0 pip-7.1.2 pip-accel-0.43 pkginfo-1.9.6 property-manager-3.0 py2deb-5.0 six-1.16.0 verboselogs-1.7

[notice] A new release of pip is available: 7.1.2 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
(.venv) vitex@exiv:~/Projects/Packaging/Python$ pip install --upgrade pip
Traceback (most recent call last):
  File "/home/vitex/Projects/Packaging/Python/.venv/bin/pip", line 5, in <module>
    from pip import main
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip/__init__.py", line 13, in <module>
    from pip.utils import get_installed_distributions, get_prog
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip/utils/__init__.py", line 23, in <module>
    from pip._vendor import pkg_resources
  File "/home/vitex/Projects/Packaging/Python/.venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__init__.py", line 32, in <module>
    import symbol
ModuleNotFoundError: No module named 'symbol'

Any hint anyone please ?

trying to keep up pip-accel and py2deb with the latest version of pip

Hello @xolox ,

I am embarking myself in updating the underlying pip engine in pip-accel and py2deb. So far, I have whipped up this patch :

diff --git a/pip_accel/__init__.py b/pip_accel/__init__.py
index 5dc39f2..33d7f2d 100644
--- a/pip_accel/__init__.py
+++ b/pip_accel/__init__.py
@@ -70,13 +70,13 @@ from pip_accel.utils import (
 
 # External dependencies.
 from humanfriendly import concatenate, Timer, pluralize
-from pip import basecommand as pip_basecommand_module
-from pip import index as pip_index_module
-from pip import wheel as pip_wheel_module
-from pip.commands.download import DownloadCommand
-from pip.commands.install import InstallCommand
-from pip.exceptions import DistributionNotFound
-from pip.req import InstallRequirement
+from pip._internal.cli import base_command as pip_basecommand_module
+from pip._internal.index import package_finder as pip_index_module
+from pip._internal.models import wheel as pip_wheel_module
+from pip._internal.commands.download import DownloadCommand
+from pip._internal.commands.install import InstallCommand
+from pip._internal.exceptions import DistributionNotFound
+from pip._internal.req import InstallRequirement
 
 # Semi-standard module versioning.
 __version__ = '0.43'
@@ -391,7 +391,7 @@ class PipAccelerator(object):
         """
         unpack_timer = Timer()
         logger.info("Unpacking distribution(s) ..")
-        with PatchedAttribute(pip_basecommand_module, 'PackageFinder', CustomPackageFinder):
+        with PatchedAttribute(pip_index_module, 'PackageFinder', CustomPackageFinder):
             requirements = self.get_pip_requirement_set(arguments, use_remote_index=False, use_wheels=use_wheels)
             logger.info("Finished unpacking %s in %s.", pluralize(len(requirements), "distribution"), unpack_timer)
             return requirements
@@ -722,7 +722,7 @@ class SetupRequiresPatch(object):
         """Enable caching of setup requirements (by patching the ``run_egg_info()`` method)."""
         if self.patch is None:
             created_links = self.created_links
-            original_method = InstallRequirement.run_egg_info
+            original_method = InstallRequirement.metadata
             shared_directory = self.config.eggs_cache
 
             def run_egg_info_wrapper(self, *args, **kw):
@@ -743,7 +743,7 @@ class SetupRequiresPatch(object):
                 return original_method(self, *args, **kw)
 
             # Install the wrapper method for the duration of the context manager.
-            self.patch = PatchedAttribute(InstallRequirement, 'run_egg_info', run_egg_info_wrapper)
+            self.patch = PatchedAttribute(InstallRequirement, 'metadata', run_egg_info_wrapper)
             self.patch.__enter__()
 
     def __exit__(self, exc_type=None, exc_value=None, traceback=None):
diff --git a/pip_accel/req.py b/pip_accel/req.py
index ad2f61b..ae8a1a9 100644
--- a/pip_accel/req.py
+++ b/pip_accel/req.py
@@ -38,7 +38,7 @@ from pip_accel.utils import hash_files
 
 # External dependencies.
 from cached_property import cached_property
-from pip.req import InstallRequirement
+from pip._internal.req import InstallRequirement
 
 # The following package(s) are usually bundled with pip but may be unbundled
 # by redistributors and pip-accel should handle this gracefully.
diff --git a/pip_accel/utils.py b/pip_accel/utils.py
index 4b168c9..f14b53a 100644
--- a/pip_accel/utils.py
+++ b/pip_accel/utils.py
@@ -25,7 +25,7 @@ from pip_accel.compat import pathname2url, urljoin, WINDOWS
 
 # External dependencies.
 from humanfriendly import parse_path
-from pip.commands.uninstall import UninstallCommand
+from pip._internal.commands.uninstall import UninstallCommand
 
 # The following package(s) are usually bundled with pip but may be unbundled
 # by redistributors and pip-accel should handle this gracefully.
diff --git a/requirements.txt b/requirements.txt
index c344326..a579c41 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,5 @@
 cached-property >= 0.1.5
 coloredlogs >= 3.0
 humanfriendly >= 1.44.7
-pip >= 8.1, < 8.2
+pip
 setuptools >= 7.0

but then, py2deb returns this error :

Traceback (most recent call last):
  File "/home/artynet/.local/lib/python3.7/site-packages/py2deb/cli.py", line 200, in main
    archives, relationships = converter.convert(arguments)
  File "/home/artynet/.local/lib/python3.7/site-packages/py2deb/converter.py", line 735, in convert
    self.packages_to_convert = list(self.get_source_distributions(pip_install_arguments))
  File "/home/artynet/.local/lib/python3.7/site-packages/py2deb/converter.py", line 792, in get_source_distributions
    for requirement in self.pip_accel.get_requirements(arguments):
  File "/home/artynet/.local/lib/python3.7/site-packages/pip_accel/__init__.py", line 299, in get_requirements
    return self.unpack_source_dists(arguments, use_wheels=use_wheels)
  File "/home/artynet/.local/lib/python3.7/site-packages/pip_accel/__init__.py", line 395, in unpack_source_dists
    requirements = self.get_pip_requirement_set(arguments, use_remote_index=False, use_wheels=use_wheels)
  File "/home/artynet/.local/lib/python3.7/site-packages/pip_accel/__init__.py", line 503, in get_pip_requirement_set
    command = DownloadCommand()
  File "/home/artynet/.local/lib/python3.7/site-packages/pip/_internal/commands/download.py", line 41, in __init__
    super(DownloadCommand, self).__init__(*args, **kw)
TypeError: __init__() missing 2 required positional arguments: 'name' and 'summary'

basically I can't figure out the right function the get the egg info. Any help would be appreciated...thanks in advance..

Arturo

Requirements limit pip to versions <7.2

When installing via pip, pip is downgraded to version 7.1.2 because the requirements limit it to be below version 7.2. The current version of pip is 9.0.1 and I think that should be supported.

$ pip3 install py2deb
Requirement already satisfied: py2deb in /usr/local/lib/python3.4/dist-packages
Requirement already satisfied: pip-accel>=0.25 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: six>=1.6.1 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: pkginfo>=1.1 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: coloredlogs>=0.5 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: humanfriendly>=2.3.1 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: cached-property>=0.1.5 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: deb-pkg-tools>=1.30 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: executor>=1.2 in /usr/local/lib/python3.4/dist-packages (from py2deb)
Requirement already satisfied: setuptools>=7.0 in /usr/local/lib/python3.4/dist-packages (from pip-accel>=0.25->py2deb)
Collecting pip<7.2,>=7.0 (from pip-accel>=0.25->py2deb)
  Using cached pip-7.1.2-py2.py3-none-any.whl
Requirement already satisfied: chardet in /usr/local/lib/python3.4/dist-packages (from deb-pkg-tools>=1.30->py2deb)
Requirement already satisfied: python-memcached>=1.58 in /usr/local/lib/python3.4/dist-packages (from deb-pkg-tools>=1.30->py2deb)
Requirement already satisfied: python-debian>=0.1.21-nmu2 in /usr/lib/python3/dist-packages (from deb-pkg-tools>=1.30->py2deb)
Requirement already satisfied: property-manager>=2.1 in /usr/local/lib/python3.4/dist-packages (from deb-pkg-tools>=1.30->py2deb)
Requirement already satisfied: fasteners>=0.14.1 in /usr/local/lib/python3.4/dist-packages (from executor>=1.2->py2deb)
Requirement already satisfied: verboselogs>=1.1 in /usr/local/lib/python3.4/dist-packages (from property-manager>=2.1->deb-pkg-tools>=1.30->py2deb)
Requirement already satisfied: monotonic>=0.1 in /usr/local/lib/python3.4/dist-packages (from fasteners>=0.14.1->executor>=1.2->py2deb)
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-7.1.2
You are using pip version 7.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

pyinstaller wrapper support..

would be nice to run pyinstaller while building the deb... exe > deb done..
script to executable then deb-pack..

https://bitbucket.org/jan_de_bleser/openport-client/src

sure he has x86/x64 debs/rpms msi-win.. , however Arm/arm64 not so much...
useful for runing pi'3 or arm64 servers over ssh tunnels but its being rather a pain in the backside to .... build as wanted.

Permissions error

Hello I'm running py2deb -- . and getting an error.

py2deb 2.1
Python 3.7.3

I'm not sure which directory it's referring to but my project directory has permissions 755.

deb_pkg_tools.package[25587] INFO Building package in /tmp/py2deb-build-3itgtat_ ..
dpkg-deb: error: control directory has bad permissions 750 (must be >=0755 and <=0775)
Traceback (most recent call last):
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/py2deb/cli.py", line 200, in main
    archives, relationships = converter.convert(arguments)
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/py2deb/converter.py", line 752, in convert
    archive = package.convert()
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/py2deb/package.py", line 502, in convert
    copy_files=False)
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/deb_pkg_tools/package.py", line 718, in build_package
    fakeroot=ALLOW_FAKEROOT_OR_SUDO, logger=logger)
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/executor/__init__.py", line 174, in execute
    return execute_prepared(ExternalCommand(*command, **options))
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/executor/__init__.py", line 203, in execute_prepared
    command.start()
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/executor/__init__.py", line 1441, in start
    self.start_once(**kw)
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/executor/__init__.py", line 1508, in start_once
    self.wait(check=check)
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/executor/__init__.py", line 1551, in wait
    self.check_errors(check=check)
  File "/home/user/Documents/mypackage/venv/lib/python3.7/site-packages/executor/__init__.py", line 1673, in check_errors
    raise self.error_type(self)
executor.ExternalCommandFailed: External command failed with exit code 2!

Command:
fakeroot dpkg-deb --build /tmp/py2deb-build-3itgtat_ /tmp/deb-pkg-tools-build-90n472fs/python3-alabaster_0.7.12_all.deb

packages with pth-files can't be build

When trying to build coloredlogs, I encountered the error

Expected to find a single 'dist-packages' directory inside converted package!

when building with --install-prefix=/usr/local. The second dist-package py2deb encounters contains the pth-file of coloredlogs and AFAIK it's correct that it should point to a different directory (here /usr/local/lib/python3/dist-packages/ instead of /usr/local/lib/python3.5/dist-packages/coloredlogs).

please install module in order to install this package

I get such error if I run:

~/.local/bin/py2deb -r /tmp -- buildbot-www

2020-04-24 12:37:49 lts-debug pip.utils[24552] INFO Complete output from command python setup.py egg_info:
2020-04-24 12:37:49 lts-debug pip.utils[24552] INFO Please install buildbot_pkg module in order to install that package, or use the pre-build .whl modules available on pypi

----------------------------------------
2020-04-24 12:37:49 lts-debug py2deb.cli[24552] ERROR Caught an unhandled exception!

So the problem that buildbot-www is depend on buildbot.
I create requirements.txt:
requirements.txt.zip
with buildbot and buildbot-www and all dependicies:

$ grep buildbot requirements.txt
buildbot==2.7.0
buildbot-www==2.7.0

but I got the same error during:

 ~/.local/bin/py2deb -r /tmp -- -r /tmp/requirements.txt

So how can I can I help py2deb find buildbot package?

Generate a .changes file for uploading to mentors.debian.net

Hi!

Thanks so much for this tool, it is really helpful.
I've been able to generate a .deb file for my python package and .deb files for all of its dependencies (all of which are written to /tmp), but I'm struggling to create a *.changes file to upload to mentors.debian.net. It seems like dput, the tool used to upload, will only accept a *.changes file. Is there a way to do that with this tool, or is there a process you know of that can take the output from this tool and turn it into a .changes file?

Thanks!

Small doc change

There's a snippet of code on http://py2deb.readthedocs.io/en/stable/developers.html?highlight=post#module-py2deb.converter that currently reads:

>>> from py2deb import PackageConverter
>>> converter = PackageConverter()
>>> archives, relationships = converter.convert(['py2deb'])
>>> print(archives)
['/tmp/python-py2deb_0.18_all.deb']
>>> print(relationships)
['python-py2deb (=0.18)']

I'm pretty sure that first line should be

>>> from py2deb.converter import PackageConverter

Upgrade pip?

Why does pip need to be downgraded to 7.x, when the latest is 10.0.1? 🤔

Question about post-installation

I'm trying to add a post installation script with either the py2deb command line or the py2deb python API. I'm looking at the following resources and I'm not quite 'getting it'. My specific use case is I need to add postinst functionality to the debian package (the typical small shell script).... can someone point me in the right direction? :)

Resources I'm looking at so far:

• http://py2deb.readthedocs.io/en/stable/developers.html?highlight=post#module-py2deb.converter
• http://www.manatlan.com/page/py2deb
• http://www.manatlan.com/page/py2deb_configuration
• https://wiki.maemo.org/Py2deb

Support for wheel distributions

With the increasing adoption of the wheel distribution archive format in the Python community in the past few years, I've noticed package authors starting to publish new releases to PyPI as wheels exclusively (i.e. without an accompanying source distribution):

• One example of where I ran into this was textX 1.5.1.
• I'd previously run into this with email_validator 1.0.3.

While writing up this issue text I notice that a more recent version of textX has restored source distribution uploads, but email_validator hasn't. More importantly this is not about specific examples but about the general trend.

Upgrading the version of pip used by pip-accel

Lately I've considered the biggest blocker for this (adding wheel support to py2deb) to be upgrading pip-accel to use a newer version of pip:

• The version of pip used by pip-accel is currently pinned to 7.1.2, see also #14.
• While pip-accel with pip 7.1.2 does support wheels, the pip-accel API doesn't expose it. Furthermore adding full-fledged support for wheels to py2deb based on pip 7.1.2 now and then upgrading pip-accel and py2deb to use pip 9.x may duplicate a lot of work.
• I could decide that pip-accel is a dying project that's not worth saving (this touches on paylogic/pip-accel#78) but then I'd have to rewrite py2deb to work without pip-accel, which will be a major undertaking as well. I'm hesitant to make that move and would rather improve pip-accel instead.

Minimize scope by skipping pip-accel upgrade?

Entertaining the thought of adding wheel support to py2deb without involving a pip-accel upgrade (because scope creep):

• Wheel is a format with a proper specification so implementing support for wheels in py2deb now and upgrading pip and pip-accel later might not duplicate much work.
• While py2deb needs to understand wheel distributions in order to transform them into Debian packages, pip-accel just needs to hand them off to pip for installation.

UnicodeDecodeError when attrs in requirements.txt

I'm attempting to use py2deb for a python 3 project and I run into the following error:

root@547626b5656f:/test# py2deb -r /tmp -- -r requirements.txt 
2018-06-26 13:00:01 547626b5656f pip_accel[7224] INFO Unpacking distribution(s) ..
2018-06-26 13:00:01 547626b5656f pip_accel[7224] INFO Executing command: pip install --download=/var/cache/pip-accel/sources --find-links=file:///var/cache/pip-accel/sources --no-binary=:all: --no-index --no-clean --build-directory=/tmp/pip-accel-build-dir-pfxjgqit --ignore-installed -r requirements.txt
2018-06-26 13:00:01 547626b5656f pip.commands.install[7224] INFO Ignoring indexes: https://pypi.python.org/simple
2018-06-26 13:00:01 547626b5656f pip.req.req_set[7224] INFO Collecting attrs==18.1.0 (from -r requirements.txt (line 1))
2018-06-26 13:00:01 547626b5656f pip.download[7224] INFO File was already downloaded /var/cache/pip-accel/sources/attrs-18.1.0.tar.gz
2018-06-26 13:00:02 547626b5656f pip.commands.install[7224] INFO Successfully downloaded attrs
2018-06-26 13:00:02 547626b5656f pip_accel[7224] INFO Finished unpacking 1 distribution in 0.32 seconds.
2018-06-26 13:00:02 547626b5656f py2deb.cli[7224] ERROR Caught an unhandled exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pkginfo/sdist.py", line 70, in read
    return f.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4233: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/py2deb/cli.py", line 200, in main
    archives, relationships = converter.convert(arguments)
  File "/usr/local/lib/python3.5/dist-packages/py2deb/converter.py", line 546, in convert
    archive = package.convert()
  File "/usr/local/lib/python3.5/dist-packages/py2deb/package.py", line 415, in convert
    maintainer=self.debian_maintainer,
  File "/usr/local/lib/python3.5/dist-packages/cached_property.py", line 35, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python3.5/dist-packages/py2deb/package.py", line 130, in debian_maintainer
    maintainer = self.metadata.maintainer
  File "/usr/local/lib/python3.5/dist-packages/cached_property.py", line 35, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python3.5/dist-packages/py2deb/package.py", line 169, in metadata
    return UnpackedSDist(self.find_egg_info_file())
  File "/usr/local/lib/python3.5/dist-packages/pkginfo/sdist.py", line 65, in __init__
    filename, metadata_version=metadata_version)
  File "/usr/local/lib/python3.5/dist-packages/pkginfo/sdist.py", line 12, in __init__
    self.extractMetadata()
  File "/usr/local/lib/python3.5/dist-packages/pkginfo/distribution.py", line 108, in extractMetadata
    data = self.read()
  File "/usr/local/lib/python3.5/dist-packages/pkginfo/sdist.py", line 73, in read
    % (self.filename, e))
ValueError: Could not load /tmp/pip-accel-build-dir-pfxjgqit/attrs/pip-egg-info/attrs.egg-info/ as an unpacked sdist: 'ascii' codec can't decode byte 0xe2 in position 4233: ordinal not in range(128)

To reproduce make an empty project that just has the requirements.txt and no python code at all.

The contents of requirements.txt:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
attrs==18.1.0

And for reference:

root@547626b5656f:/test# pip -V
pip 7.1.2 from /usr/local/lib/python3.5/dist-packages (python 3.5)

Attempting the same but for one of the other requirements such as pymongo==3.5.1 works fine.

This was run inside an Ubuntu 16.04 and a Ubuntu 18.04 docker image.

The build/install steps inside the container:

mkdir builds
apt update
apt install -y python3-pip dpkg-dev fakeroot libpq-dev
pip3 install py2deb
py2deb -r ./builds -- -r requirements.txt

embed_install_prefix produces empty files when a custom install prefix is used

I noticed that, when using a custom install prefix, the .deb packages generated by py2deb contain empty files in the <install-prefix>/bin/ folder. For example, when using py2deb to build a uwsgi package, the uwsgi binary is an empty file in the resulting package.

I tracked this error to py2deb.package.PackageToConvert.transform_binary_dist. If a custom install prefix is used, this function calls py2deb.utils.embed_install_prefix on files in the bin/ folder. embed_install_prefix inspects the contents of these files to find out if they are Python executables and, if so, injects some code into them. The problem occurs only for files that are not Python executables (i.e. they lack the respective shebang). The contents of those files are read from the handle parameter, then the function immediately returns. As handle is a file-like object, this reading of the contents advances the file pointer to the end of the file. Later, handle is read again to copy the file to its final destination in the .deb package, but as the file pointer is still at EOF, the resulting file is empty.

To fix this, we could simply reset the file pointer of handle at the end of embed_install_prefix.

Library cannot be found

I got an error while trying to build a debian package for pytorch, which is due to problem in detecting the shared libraries installed with the package (in the lib directory).

dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libshm.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: error: couldn't find library libATen.so.1 needed by /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libshm.so (ELF format: 'elf64-x86-64'; RPATH: '$ORIGIN')
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libshm.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libshm.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libshm.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: couldn't find library libshm.so needed by /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so (ELF format: 'elf64-x86-64'; RPATH: '$ORIGIN/lib')
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: error: couldn't find library libATen.so.1 needed by /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so (ELF format: 'elf64-x86-64'; RPATH: '$ORIGIN/lib')
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: $ORIGIN is used in RPATH of /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so and the corresponding directory could not be identified due to lack of DEBIAN sub-directory in the root of package's build tree
dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
dpkg-shlibdeps: error: cannot continue due to the errors listed above
Note: libraries are not searched in other binary packages that do not have any shlibs or symbols file.
To help dpkg-shlibdeps find private libraries, you might need to use -l.2018-04-04 16:25:35 linuxmint17 py2deb.cli[10899] ERROR Caught an unhandled exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/py2deb/cli.py", line 200, in main
    archives, relationships = converter.convert(arguments)
  File "/usr/local/lib/python3.4/dist-packages/py2deb/converter.py", line 546, in convert
    archive = package.convert()
  File "/usr/local/lib/python3.4/dist-packages/py2deb/package.py", line 409, in convert
    dependencies += self.find_system_dependencies(shared_object_files)
  File "/usr/local/lib/python3.4/dist-packages/py2deb/package.py", line 561, in find_system_dependencies
    output = execute(*command, directory=fake_source_directory, capture=True, logger=logger)
  File "/usr/local/lib/python3.4/dist-packages/executor/__init__.py", line 174, in execute
    return execute_prepared(ExternalCommand(*command, **options))
  File "/usr/local/lib/python3.4/dist-packages/executor/__init__.py", line 203, in execute_prepared
    command.start()
  File "/usr/local/lib/python3.4/dist-packages/executor/__init__.py", line 1392, in start
    self.wait()
  File "/usr/local/lib/python3.4/dist-packages/executor/__init__.py", line 1435, in wait
    self.check_errors(check=check)
  File "/usr/local/lib/python3.4/dist-packages/executor/__init__.py", line 1553, in check_errors
    raise self.error_type(self)
executor.ExternalCommandFailed: External command failed with exit code 2!

Command:
dpkg-shlibdeps -O --warnings=0 /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_dl.cpython-34m.so /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/_C.cpython-34m.so /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libshm.so /tmp/py2deb-build-l8yv6y4p/usr/lib/python3.4/dist-packages/torch/lib/libATen.so

py2deb doesn't handle tilde-equal (~=) version specifications in requirements

This is referenced in a couple of comments in issue #8 but that seems to be a different issue. So opening this issue to address this.
I'm specifically talking about #8 (comment):

...
2017-05-22 07:51:22 v2g-dev py2deb.converter[17131] INFO Package pyserial (3.2.1) already converted: /tmp/python3-pyserial_3.2.1_all.deb
2017-05-22 07:51:22 v2g-dev py2deb.cli[17131] ERROR Caught an unhandled exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/py2deb/cli.py", line 192, in main
    archives, relationships = converter.convert(arguments)
  File "/usr/local/lib/python3.5/dist-packages/py2deb/converter.py", line 524, in convert
    archive = package.convert()
  File "/usr/local/lib/python3.5/dist-packages/py2deb/package.py", line 383, in convert
    dependencies = [python_version()] + self.debian_dependencies
  File "/usr/local/lib/python3.5/dist-packages/cached_property.py", line 26, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python3.5/dist-packages/py2deb/package.py", line 318, in debian_dependencies
    raise Exception(msg % (constraint, self.python_name))
Exception: Conversion specifier not supported! ('~=' used by Python package uModbus)