dephell / dephell

:package: :fire: Python project management. Manage packages: convert between formats, lock, install, resolve, isolate, test, build graph, show outdated, audit. Manage venvs, build package, bump version.

THE PROJECT IS ARCHIVED

Forks: https://github.com/orsinium/forks

---

DepHell -- project management for Python.

Why it is better than all other tools:

1. Format agnostic. You can use DepHell with your favorite format: setup.py, requirements.txt, Pipfile, poetry. DepHell supports them all and much more.
2. Use your favorite tool on any project. Want to install a poetry based project, but don't like poetry? Just tell DepHell to convert the project's meta information into a setup.py and install it with pip. Or directly work with the project from DepHell, because DepHell can do everything that you usually want to do with packages.
3. DepHell doesn't try to replace your favorite tools. If you use poetry, you have to use poetry's file formats and commands. However, DepHell can be combined with any other tool or can even combine all these tools together by converting formats. You can use DepHell, poetry, and pip all at the same time.
4. Easily extendable. DepHell has strong modularity and can be easily extended with new formats and commands.
5. Developer friendly. We aren't going to place all of our modules into _internal. Also, DepHell has a large ecosystem with separated libraries so you can use only the parts of DepHell that you need.
6. All-in-one-solution. DepHell can manage dependencies, virtual environments, tests, CLI tools, packages, generate configs, show licenses for dependencies, assist with security audits, get download statistics from PyPI, search packages and much more.
7. Smart dependency resolution. Sometimes pip and pipenv fail to lock your dependencies. Try to execute pipenv install oslo.utils==1.4.0. Pipenv can't handle it, but DepHell can: dephell deps add --from=Pipfile oslo.utils==1.4.0 to add new dependency and dephell deps convert --from=Pipfile --to=Pipfile.lock to lock it.
8. Asyncio based. DepHell doesn't support Python 2.7, and that allows us to use modern features to make network and filesystem requests as fast as possible.
9. Multiple environments. You can have as many environments for project as you want. Separate sphinx dependencies from your main and dev environment. Other tools like pipenv and poetry don't support this.

Features:

• Manage dependencies: convert between formats, instаll, lock, add new, resolve conflicts.
• Manage virtual environments: create, remove, inspect, run shell, run commands inside.
• Install CLI tools into isolated environments.
• Manage packages: install, list, search on PyPI.
• Build packages (to upload on PyPI), test, bump project version.
• Discover licenses of all project dependencies, show outdated packages, find security issues.
• Generate .editorconfig, LICENSE, AUTHORS, .travis.yml.

See documentation for more details.

Follow @PythonDepHell on Twitter to get updates about new features and releases.

Installation

curl -L dephell.org/install | python3

See installation documentation for alternatives.

Supported formats

1. Archives:
   1. *.egg-info (egginfo)
   2. *.tar.gz (sdist)
   3. *.whl (wheel)
2. pip:
   1. requirements.txt (pip)
   2. requirements.lock (piplock)
3. pipenv:
   1. Pipfile (pipfile)
   2. Pipfile.lock (pipfilelock)
4. pоetry:
   1. pyproject.toml (poetry)
   2. poetry.lock (poetrylock)
5. Environment:
   1. Imports in the package (imports).
   2. Installed packages (installed).
6. Other:
   1. setup.py (setuppy)
   2. flit (flit)
   3. conda's environment.yml (conda)
   4. pyproject.toml build-system requires (pyproject)

Usage

First of all, install DepHell and activate autocomplete:

python3 -m pip install --user dephell[full]
dephell self autocomplete

Let's get sampleproject and make it better.

git clone https://github.com/pypa/sampleproject.git
cd sampleproject

This project uses setup.py for dependencies and metainfo. However, this format is over-complicated for most projects. Let's convert it into poetry:

dephell deps convert --from=setup.py --to=pyproject.toml

It will make next pyproject.toml:

[tool.poetry]
name = "sampleproject"
version = "1.2.0"
description = "A sample Python project"
authors = ["The Python Packaging Authority <[email protected]>"]
readme = "README.md"

[tool.poetry.scripts]
sample = "sample:main"

[tool.poetry.dependencies]
python = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,<4,>=2.7"
coverage = {optional = true}
peppercorn = "*"

[tool.poetry.dev-dependencies]
check-manifest = "*"

[tool.poetry.extras]
test = ["coverage"]

Now, let's generate some useful files:

dephell generate authors

dephell generate license MIT

# https://editorconfig.org/
dephell generate editorconfig

Our users probably have not installed poetry, but they are likely to have pip and can install files from setup.py. Let's make it easier to generate setup.py from our pyproject.toml. Also, it points to DepHell as your default dependencies file. Add the following lines in the pyproject.toml:

[tool.dephell.main]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "setuppy", path = "setup.py"}

You can see a full, real-world example of a config in DepHell's own pyproject.toml.

Now we can call DepHell commands without explicitly specifying from and to:

dephell deps convert

It will make setup.py and README.rst from pyproject.toml and README.md.

Now let's test our code in a virtual environment:

$ dephell venv run pytest
WARNING venv does not exist, creating... (project=/home/gram/Documents/sampleproject, env=main, path=/home/gram/.local/share/dephell/venvs/sampleproject-Whg0/main)
INFO venv created (path=/home/gram/.local/share/dephell/venvs/sampleproject-Whg0/main)
WARNING executable does not found in venv, trying to install... (executable=pytest)
INFO build dependencies graph...
INFO installation...
# ... pip output
# ... pytest output

We can now activate the virtual environment for our project and run any commands inside:

dephell venv shell

Ugh, we have tests, but don't have pytest in our dependencies file. Let's add it:

dephell deps add --envs dev test -- pytest

Afer that our dev-dependencies looks like this:

[tool.poetry.dev-dependencies]
check-manifest = "*"
pytest = "*"

[tool.poetry.extras]
test = ["coverage", "pytest"]

Eventually we will have many more dependencies. Let's look at how many of them we have now:

$ dephell deps tree
- check-manifest [required: *, locked: 0.37, latest: 0.37]
- coverage [required: *, locked: 4.5.3, latest: 4.5.3]
- peppercorn [required: *, locked: 0.6, latest: 0.6]
- pytest [required: *, locked: 4.4.0, latest: 4.4.0]
  - atomicwrites [required: >=1.0, locked: 1.3.0, latest: 1.3.0]
  - attrs [required: >=17.4.0, locked: 19.1.0, latest: 19.1.0]
  - colorama [required: *, locked: 0.4.1, latest: 0.4.1]
  - funcsigs [required: >=1.0, locked: 1.0.2, latest: 1.0.2]
  - more-itertools [required: <6.0.0,>=4.0.0, locked: 5.0.0, latest: 7.0.0]
    - six [required: <2.0.0,>=1.0.0, locked: 1.12.0, latest: 1.12.0]
  - more-itertools [required: >=4.0.0, locked: 7.0.0, latest: 7.0.0]
  - pathlib2 [required: >=2.2.0, locked: 2.3.3, latest: 2.3.3]
    - scandir [required: *, locked: 1.10.0, latest: 1.10.0]
    - six [required: *, locked: 1.12.0, latest: 1.12.0]
  - pluggy [required: >=0.9, locked: 0.9.0, latest: 0.9.0]
  - py [required: >=1.5.0, locked: 1.8.0, latest: 1.8.0]
  - setuptools [required: *, locked: 41.0.0, latest: 41.0.0]
  - six [required: >=1.10.0, locked: 1.12.0, latest: 1.12.0]

Hm...Is it as many as it seems? Let's look at their size.

$ dephell inspect venv --filter=lib_size
11.96Mb

Ugh...Ok, it's Python. Are they actual?

$ dephell deps outdated
[
  {
    "description": "More routines for operating on iterables, beyond itertools",
    "installed": [
      "5.0.0"
    ],
    "latest": "7.0.0",
    "name": "more-itertools",
    "updated": "2019-03-28"
  },
]

Pytest requires old version of more-itertools. That happens.

If our tests and dependencies are OK, it's time to deploy. First of all, increment the project version:

$ dephell project bump minor
INFO generated new version (old=1.2.0, new=1.3.0)

And then build packages:

$ dephell project build
INFO dumping... (format=setuppy)
INFO dumping... (format=egginfo)
INFO dumping... (format=sdist)
INFO dumping... (format=wheel)
INFO builded

Now, we can upload these packages to PyPI:

dephell self auth upload.pypi.org my-username my-password
dephell project upload

These are some of the most useful commands. See documentation for more details.

Compatibility

DepHell is tested on Linux and Mac OS X with Python 3.5, 3.6, 3.7. And one of the coolest things is that DepHell is run by DepHell on Travis CI.

DepHell works on Windows but has no CI to keep in working yet.

How can I help

1. Star project on Github. Developers believe in the stars.
2. Tell your fellows that Gram has a made cool thing for you.
3. Open an issue if you have thoughts on how to make DepHell better.
4. Things that you can contribute in any project in DepHell ecosystem:
   1. Fix grammar and typos.
   2. Document new things.
   3. Tests, we always need more tests.
   4. Make READMEs more nice and friendly.
   5. View issues with the help wanted label to find things that you can fix.
   6. Anything what you want. If it is a new feature, please, open an issue before writing the code.

Thank you ❤️

dephell's Issues

dot in package name gets converted to - during convert

I was converting poetry project to setuppy and pip. On both, dependency with a dot in name (ex. ruamel.yaml) have its dot replaced by -. Which is wrong for further installations

`build` command

Just creates setup.py (if it's not the source), wheel, sdist.

Package search command

`jail update` command

Update all packages in a single jail or all jails.

bump version

`add` command

`jail show` command

Command to show all useful information about jail. Like inspect venv, but with exposed entrypoints list.

Github API support

Now DepHell downloads all repository for VCS-based dependency and makes checkout for every project version to get actual dependencies. It's a really difficult task for big projects. We can do it better by requesting Github API.

document CLI args

Custom inline dependency format

Now DepHell commands read dependencies in pip format. Example:

dephell package install "-e dephell[full]"

We need a new modern universal format, readable and intuituve.

Initial idea: posix-compatible options.

dephell package install "dephell --editable --extra full" 

Pip can't work hashes when project depends on local path

1. If hashes are presented pip forces hashes usage for all deps.
2. pip can't calculate hashes for dir.

Can't verify hashes for these file:// requirements because they point to directories:

So, if there are dir based deps we have to disable hashes for piplock

`dephell project cd` command

Store all projects (directories where dephell was runned with config) paths in the textfile in dephell's datadir and perform search for the best path by command

$ dephell project cd test

~/local/dephell/paths.txt:

/home/gram/Documents/lol/
/home/gram/Documents/test/
/home/gram/Documents/nani/

In this example dephell will cd into /home/gram/Documents/test/

Drop html2text usage

html2text distributed under GNU GPL v3.0, but DepHell uses MIT. This is incompatible. I'm using html2text to generate text for a terminal with Jinja2 help. If generate simple text with Jinja2 then this text will have some extra empty lines where Jinja2 operators were. Does anyone know other solutions for this?

dephell build-backend

Let's make PEP-518 build-backend to install dependencies from the main env.

'dephell autocomplete' fails when installing it into pyenv-managed virtualenv

On my development environment I'm using pyenv, pyenv-virtualenv and zsh. So, when I'm doing 'pip install dephell' inside virtualenv based on pyenv shim and running 'dephell autocomplete' afterwards, it gives me something like "_ERROR: Can't find '/home/meow-nofer/.local/share/pyenv/dephell_zsh_autocompletion'".

Okay then, so I installed it without virtualenv, just inside my pyenv's shim. Now I'm getting this:

Traceback (most recent call last):
  File "/home/meow-nofer/.local/bin/dephell", line 6, in <module>
    from dephell.cli import entrypoint
  File "/home/meow-nofer/.local/lib/python3.7/site-packages/dephell/__init__.py", line 3, in <module>
    from .controllers import *  # noQA
  File "/home/meow-nofer/.local/lib/python3.7/site-packages/dephell/controllers/__init__.py", line 3, in <module>
    from .dependency import DependencyMaker
  File "/home/meow-nofer/.local/lib/python3.7/site-packages/dephell/controllers/dependency.py", line 8, in <module>
    from dephell_links import VCSLink, parse_link
  File "/home/meow-nofer/.local/lib/python3.7/site-packages/dephell_links/__init__.py", line 2, in <module>
    from ._parse import parse_link
  File "/home/meow-nofer/.local/lib/python3.7/site-packages/dephell_links/_parse.py", line 5, in <module>
    from ._vcs import VCSLink
  File "/home/meow-nofer/.local/lib/python3.7/site-packages/dephell_links/_vcs.py", line 9, in <module>
    @attr.s()
AttributeError: module 'attr' has no attribute 's'```

cached-property dependency

it seems that https://pypi.org/project/cached-property/ is missing in en dependency list, i have to install it manually

python3 -m pip install dephell[all] cached-property

Improve installation on clean system

1. add setuptools into requirements
2. install wheel into venv
3. check that python3-venv installed

TTL for cache

Include dependency_links when converting git dependencies

I have a pyproject.toml file with a git dependency:
libtest = {git = "https://github.com/gwtwod/poetrylibtest.git", branch = "master"}

When converting to setup.py the URL information is lost, example:
dephell deps convert --from-format=poetry --from-path=pyproject.toml --to-format=setuppy --to-path=setup.py

I get a setup.py with this which wont be able to install since the package is not on pypi:
install_requires=['libtest']

If I then manually add this section to setup.py with the egg name it works as expected:

dependency_links=[
    'git+https://github.com/gwtwod/poetrylibtest#egg=libtest-0.1.0',
]

More examples for `deps convert` command

https://github.com/dephell/dephell/blob/master/docs/cmd-deps-convert.md gives lots of examples but not one of the form:

dephell deps convert --from=requirements.txt --to=pyproject.toml

Debian repositories support

Debian has search api and package info api. We can use it for show, search and releases commands.

check command

to check is environment is actual with lockfile or best_version

Python 3.5.2 support

Aiohttp requires at least python 3.5.4. Could we use something else or old aiohttp version?

Support of custom warehouse

Now dephell uses a new pypi api to get information about packages. A few simple manipulations will allow us to use any custom repository.

allow to specify path to `.env` file

For example:

dephell venv run --dotenv=prod.env.sh

Patch:

        env_vars = read_dotenv(
            path=Path(self.config.get('dotenv') or self.config['project']),
            env_vars=env_vars,
        )

`jail inject` command to install packages into existing jail

For example, dephell jail inject flake8 flake8-logging to install package into flake8 jail.

`get` command that works like pipsi

Because I can

Reversed dependencies for a package

Reversed dependencies are really helpful and interesting information. However, I'm not sure how we can implement it.

• libraries.io knows it, but their API requires authtorization, and we don't want to force our users register some accounts for DepHell usage.
• pypi can't implement it because they don't really know real projects dependencies.

sync command

to remove all old packages before deps installation

`package upload` command

Correct use pathlib everywhere in tests

For example, instead of

'./tests/requirements/attrs-requests.txt'

We must write

Path() / 'tests' / 'requirements' / 'attrs-requests.txt'

This is the first required step to test DepHell on Windows.

Tox support

Document graph building

Function name typo

The analize_conflict function should be named analyze_conflict.
https://github.com/dephell/dephell/blob/master/dephell/controllers/conflict.py#L43

document commands

Package show command

Flit support

`test` command

Run pytest on all Pythons that project support in the separated venvs.

`install` command

'python_requires' invalid version specifiers

hi,

i encountered this issue while using another tool on the setup.py file generated using dephell (stdeb with debianize command)

'python_requires' must be a string containing valid version specifiers; Invalid specifier: '>=2.7.0 || ==3.*'

i checked and that specifier doesn't seem correct (no OR in the specification of versions)

quick run some package into tmp jail

Less dependencies

Firstly, let's reuse _vendor dirs from our deps to get appdirs, packaging and requests. Maybe, something more.

graph command

Script to install dephell

it will be cool to make a script that installs dephell into dephell's venv path to isolate all dephell's dependencies from global modules.

Test untested commands

A few commands are untested yet. We need smoke tests for them like for other commands. I think it is suitable for the first issue because we already have tests for much of commands, and all they are similar.

Docker integration

We can use docker instead of venv, it's quite simple. See pydockenv for example. Unfortunately, this lib doesn't provide any API, so let's reimplement it.

PEP-582 support

PEP-582 is a kind of node_modules for python. It installs project dependencies into __pypackages__ directory in the project root. It's like venv without binaries. Should we support it?

One simple implementation: pythonloc

ImportError when setup.py tries to import local modules

We are trying to use dephell deps tree command with --from=setup.py for projects ipython or jupyter_console and it does not work.

We are getting the following errors:

• for ipython

(dephell)$ dephell deps tree --from=/path/to/ipython/setup.py --traceback
WARNING cannot find config file 
ERROR ImportError: No module named 'setupbase' 
ERROR distribution was not called 
ERROR ImportError: No module named 'setupbase' 
Traceback (most recent call last):
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/cli.py", line 75, in main
    result = task()
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/commands/deps_tree.py", line 40, in __call__
    resolver = loader.load_resolver(path=self.config['from']['path'])
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/converters/base.py", line 69, in load_resolver
    root = self.load(path)
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/converters/setuppy.py", line 83, in load
    info = run_setup(path.name)
  File "/usr/lib/python3.5/distutils/core.py", line 216, in run_setup
    exec(f.read(), g, l)
  File "<string>", line 69, in <module>
ImportError: No module named 'setupbase'

• for jupyter_console

(dephell)$ dephell deps tree --from=/path/to/jupyter_console/setup.py --traceback
WARNING cannot find config file 
ERROR SyntaxError: invalid syntax (<string>, line 1) 
ERROR distribution was not called 
ERROR RuntimeError: 'distutils.core.setup()' was never called -- perhaps 'setup.py' is not a Distutils setup script? 
Traceback (most recent call last):
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/cli.py", line 75, in main
    result = task()
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/commands/deps_tree.py", line 40, in __call__
    resolver = loader.load_resolver(path=self.config['from']['path'])
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/converters/base.py", line 69, in load_resolver
    root = self.load(path)
  File "/home/espdev/workspace/venvs/dephell/lib/python3.5/site-packages/dephell/converters/setuppy.py", line 83, in load
    info = run_setup(path.name)
  File "/usr/lib/python3.5/distutils/core.py", line 228, in run_setup
    script_name)
RuntimeError: 'distutils.core.setup()' was never called -- perhaps 'setup.py' is not a Distutils setup script?

How to use deps convert for split dev/main requirements?

I have pyproject.toml with [tool.poetry.dependencies] and [tool.poetry.dev-dependencies] blocks and try convert it with this dephell config:

[tool.dephell.main]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "pip", path = "pip-requirements.txt"}

Has dephell any possibilities for select only dev requirements convert to pip format?

No module named dephell_venvs

python -m dephell

Traceback (most recent call last):
  File "/Users/Boger/.pyenv/versions/3.7.2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/Boger/.pyenv/versions/3.7.2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/Boger/Library/Caches/pypoetry/virtualenvs/jibrel-test-assignment-py3.7/lib/python3.7/site-packages/dephell/__main__.py", line 2, in <module>
    from .cli import entrypoint
  File "/Users/Boger/Library/Caches/pypoetry/virtualenvs/jibrel-test-assignment-py3.7/lib/python3.7/site-packages/dephell/cli.py", line 8, in <module>
    from .commands import commands
  File "/Users/Boger/Library/Caches/pypoetry/virtualenvs/jibrel-test-assignment-py3.7/lib/python3.7/site-packages/dephell/commands/__init__.py", line 2, in <module>
    from .autocomplete import AutocompleteCommand
  File "/Users/Boger/Library/Caches/pypoetry/virtualenvs/jibrel-test-assignment-py3.7/lib/python3.7/site-packages/dephell/commands/autocomplete.py", line 10, in <module>
    from ..actions import make_bash_autocomplete, make_zsh_autocomplete
  File "/Users/Boger/Library/Caches/pypoetry/virtualenvs/jibrel-test-assignment-py3.7/lib/python3.7/site-packages/dephell/actions/__init__.py", line 10, in <module>
    from ._entrypoints import get_entrypoints
  File "/Users/Boger/Library/Caches/pypoetry/virtualenvs/jibrel-test-assignment-py3.7/lib/python3.7/site-packages/dephell/actions/_entrypoints.py", line 6, in <module>
    from dephell_venvs import VEnv
ModuleNotFoundError: No module named 'dephell_venvs'