pypa / pipfile

Pipfile: the replacement for requirements.txt

Pipfile and its sister Pipfile.lock are designed as a proposed replacement for an existing format: pip's requirements.txt file.

WARNING: This format is still under active design and development. Nothing is set in stone at this point in time.

This repository contains the design specification of the Pipfile format, as well as a proposed implementation of a parser for the specification which can be used by Pipenv and, in the future, any other consumer (e.g. pip), once the design (including the form of a Pipfile itself) has been built out and finalized.

---

Today, Pipenv uses Pipfile and contains the current reference implementation.

The Concept

Pipfile will be superior to requirements.txt file in a number of ways:

• TOML syntax for declaring all types of Python dependencies.
• One Pipfile (as opposed to multiple requirements.txt files).
• A Pipfile is inherently ordered.
• Existing requirements files tend to proliferate into multiple files - e.g. dev-requirements.txt, test-requirements.txt, etc. - but a Pipfile will allow seamlessly specifying groups of dependencies in one place.
  • This will be surfaced as only two built-in groups (default & development). (see note below)
• Fully specified (and deterministic) environments in the form of Pipfile.lock. A deployed application can then be completely redeployed with the same exact versions of all recursive dependencies, by referencing the Pipfile.lock file.

The concrete requirements for a Python Application would come from Pipfile. This would include where the packages should be fetched from and their loose version constraints.

The details of the environment (all installed packages with pinned versions and other details) would be stored in Pipfile.lock, for reproducibility. This file will be automatically generated and should not be modified by the user.

Note

Custom groups may be added in the future. Remember, it is easier to add features in the future than it is to remove them. The Composer community has been successful with only default and development as group options for many years. This model is being followed.

Examples (spec v6)

Here is a complex, comprehensive example Pipfile and the resulting Pipfile.lock, generated with Pipenv, and this library:

Pipfile

[[source]]
url = 'https://pypi.python.org/simple'
verify_ssl = true
name = 'pypi'

[requires]
python_version = '2.7'

[packages]
requests = { extras = ['socks'] }
records = '>0.5.0'
django = { git = 'https://github.com/django/django.git', ref = '1.11.4', editable = true }
"e682b37" = {file = "https://github.com/divio/django-cms/archive/release/3.4.x.zip"}
"e1839a8" = {path = ".", editable = true}
pywinusb = { version = "*", os_name = "=='nt'", index="pypi"}

[dev-packages]
nose = '*'
unittest2 = {version = ">=1.0,<3.0", markers="python_version < '2.7.9' or (python_version >= '3.0' and python_version < '3.4')"}

Notes:

• There will be a default source.

PEP 508 Support

# Support for all PEP 508 markers
[requires]

python_full_version = '3.6.0b1'
platform = 'windows'

requires utilizes PEP 508 marker = 'specifier' markers. This functionality may not be readily used, as it is only to assert (and therefore abort, if appropriate) installation on certain platforms (e.g. python version, platform version).

This functionality can currently be tested with $ pipenv check.

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "09da36fcc93fa9b94fbea5282d8206a9d2e13fcec27229ec62c16c134e3e760a"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "0",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.0.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64",
            "python_full_version": "2.7.14",
            "python_version": "2.7",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
                "sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5"
            ],
            "version": "==2017.7.27.1"
        },
        "chardet": {
            "hashes": [
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
            ],
            "version": "==3.0.4"
        },
        "django": {
            "editable": true,
            "git": "https://github.com/django/django.git",
            "ref": "1.11.4"
        },
        "docopt": {
            "hashes": [
                "sha256:49b3a825280bd66b3aa83585ef59c4a8c82f2c8a522dbe754a8bc8d08c85c491"
            ],
            "version": "==0.6.2"
        },
        "e1839a8": {
            "editable": true,
            "path": "."
        },
        "e682b37": {
            "file": "https://github.com/divio/django-cms/archive/release/3.4.x.zip"
        },
        "et-xmlfile": {
            "hashes": [
                "sha256:614d9722d572f6246302c4491846d2c393c199cfa4edc9af593437691683335b"
            ],
            "version": "==1.0.1"
        },
        "idna": {
            "hashes": [
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f"
            ],
            "version": "==2.6"
        },
        "jdcal": {
            "hashes": [
                "sha256:b760160f8dc8cc51d17875c6b663fafe64be699e10ce34b6a95184b5aa0fdc9e"
            ],
            "version": "==1.3"
        },
        "lxml": {
            "hashes": [
                "sha256:3593f49858fc6229cd93326be06b099ae477fd65d8f4a981320a6d0bb7fc7a5a",
                "sha256:8996df6b0f205b00b89bbd04d88f1fa1e04139a025fd291aa4ddd05dc86836f4",
                "sha256:9f399c37b8e61c3989ef12ecf0abd9c10a5075f0fc9ad1ecd67ce6f9c72a7211",
                "sha256:550a51dee73c14e5863bdbbbe5836b2b8092a3f92631b5a908b9c41e72f123a5",
                "sha256:e37eda3d05519918403084b43eb7324df21a0daf45c8ae8172a860710dd0fa78",
                "sha256:48ab0e79175fd16f9478edc679ee14c79524c64b26f665f92cbecff81312d04d",
                "sha256:52e18dd86f153c4383bb4c4ef62f81f9b7e44809d068848a5a183b2285496faf",
                "sha256:0b8f3d6e669ea26849a6184f04c7802dbef6fd418a8b90e6c026e237db07af31",
                "sha256:567b76f291a8d02aa8b4d3f8295ae749ac4d532570d8a8c7176f0556c7d95891",
                "sha256:61825daaf2d80dc3da7635ee108720b0739962db008343822753bbf343cbfd14",
                "sha256:b7f6ef610680709be11cb7662e46e254bc561dafe0de3b4848be2cf3924bd300",
                "sha256:824664493a012f5b70396e841a4b4049bdaf445a70307e60f82fe35619f72cc7",
                "sha256:e908d685800626f10cd6ae01a013fc42094be167fb2a683eb920dfddfaa0ee76",
                "sha256:10c86b2248043f4428be33ed10202764b02b281eaa4550f16f0fbbc6ccaae9ac",
                "sha256:d9ec728caddb161405e7c33ed9d176e96309893481370163bbf4b00e43008795",
                "sha256:b2ecb3fd5470b740dfc21b064bbc1337be4b7b805994a868488145d36f35f517",
                "sha256:a211288459c9115ddb255ff88e8ac12dc2063e70bddc15e3c65136477a358bb5",
                "sha256:1f81074e77c25f9b787fa3854f400ca924d3d798cb7ae910c0e7920be7138c90",
                "sha256:99b7eabfb46663ed5918eca4ed12420613ba24196964a741ccd962d09296c0b2",
                "sha256:a8ad0adeedbbb7b85916214fcd4f5d02829d0e7b3c32abc298789218b6c3d699",
                "sha256:88d137e440b5de35df2e0616da8e28a88d0119abdaa84520ad1ba815ee9da732",
                "sha256:c4e02657e629f02ab8712471d77d6896c2cf6f09f8ffa6a0f23b1b1ef0318474",
                "sha256:9581b438e5d0d0a6fa3937fac2abffd95380bd513bcd39f6952bfcf20cf0b9a7",
                "sha256:c446fde3284c363cd2085ad1ce5a07c18f15f6766d72684622bc14b0a9ddfd29",
                "sha256:d4507916c408feec2ea8cee3f0d1380e49ea431f6e07b0dd927388bd6e92d6eb",
                "sha256:7030f44b758e930fd09ade87d770f5a231a19a8c561a3acc54e5122b5ec09e29",
                "sha256:d78c0a114cf127a41a526aef99aef539c0b2537e57f04a2cc7a49e2c94a44ab8",
                "sha256:f7bc9f702500e205b1560d620f14015fec76dcd6f9e889a946a2ddcc3c344fd0"
            ],
            "version": "==4.0.0"
        },
        "odfpy": {
            "hashes": [
                "sha256:6db9bb1c9ea2d55d60e508a1318fd285442a8342b785704ea08598a260875a83",
                "sha256:6f8163f8464868cff9421a058f25566e41d73c8f7e849c021b86630941b44366"
            ],
            "version": "==1.3.5"
        },
        "openpyxl": {
            "hashes": [
                "sha256:ee7551efb70648fa8ee569c2b6a6dbbeff390cc94b321da5d508a573b90a4f17"
            ],
            "version": "==2.4.8"
        },
        "pysocks": {
            "hashes": [
                "sha256:18842328a4e6061f084cfba70f6950d9140ecf7418b3df7cef558ebb217bac8d",
                "sha256:d00329f27efa157db7efe3ca26fcd69033cd61f83822461ee3f8a353b48e33cf"
            ],
            "version": "==1.6.7"
        },
        "pytz": {
            "hashes": [
                "sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d",
                "sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9",
                "sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9",
                "sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c",
                "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
                "sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9",
                "sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043",
                "sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4",
                "sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589"
            ],
            "version": "==2017.2"
        },
        "pywinusb": {
            "hashes": [
                "sha256:e2f5e89f7b74239ca4843721a9bda0fc99014750630c189a176ec0e1b35e86df"
            ],
            "index": "pypi",
            "markers": "os_name == 'nt'",
            "version": "==0.4.2"
        },
        "pyyaml": {
            "hashes": [
                "sha256:3262c96a1ca437e7e4763e2843746588a965426550f3797a79fca9c6199c431f",
                "sha256:16b20e970597e051997d90dc2cddc713a2876c47e3d92d59ee198700c5427736",
                "sha256:e863072cdf4c72eebf179342c94e6989c67185842d9997960b3e69290b2fa269",
                "sha256:bc6bced57f826ca7cb5125a10b23fd0f2fff3b7c4701d64c439a300ce665fff8",
                "sha256:c01b880ec30b5a6e6aa67b09a2fe3fb30473008c85cd6a67359a1b15ed6d83a4",
                "sha256:827dc04b8fa7d07c44de11fabbc888e627fa8293b695e0f99cb544fdfa1bf0d1",
                "sha256:592766c6303207a20efc445587778322d7f73b161bd994f227adaa341ba212ab",
                "sha256:5f84523c076ad14ff5e6c037fe1c89a7f73a3e04cf0377cb4d017014976433f3",
                "sha256:0c507b7f74b3d2dd4d1322ec8a94794927305ab4cebbe89cc47fe5e81541e6e8",
                "sha256:b4c423ab23291d3945ac61346feeb9a0dc4184999ede5e7c43e1ffb975130ae6",
                "sha256:ca233c64c6e40eaa6c66ef97058cdc80e8d0157a443655baa1b2966e812807ca",
                "sha256:4474f8ea030b5127225b8894d626bb66c01cda098d47a2b0d3429b6700af9fd8",
                "sha256:326420cbb492172dec84b0f65c80942de6cedb5233c413dd824483989c000608",
                "sha256:5ac82e411044fb129bae5cfbeb3ba626acb2af31a8d17d175004b70862a741a7"
            ],
            "version": "==3.12"
        },
        "records": {
            "hashes": [
                "sha256:6d060a2b44ecc198d4e86efd5dab8558a2581b4019970bd8839e1604a243f57e",
                "sha256:238cba35e8efbb724493bbb195bd027d9e78db4a978597969a7af0f722ac3686"
            ],
            "version": "==0.5.2"
        },
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "version": "==2.18.4"
        },
        "sqlalchemy": {
            "hashes": [
                "sha256:f1191e29e35b6fe1aef7175a09b1707ebb7bd08d0b17cb0feada76c49e5a2d1e"
            ],
            "version": "==1.1.14"
        },
        "tablib": {
            "hashes": [
                "sha256:b8cf50a61d66655229993f2ee29220553fb2c80403479f8e6de77c0c24649d87"
            ],
            "version": "==0.12.1"
        },
        "unicodecsv": {
            "hashes": [
                "sha256:018c08037d48649a0412063ff4eda26eaa81eff1546dbffa51fa5293276ff7fc"
            ],
            "version": "==0.14.1"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        },
        "xlrd": {
            "hashes": [
                "sha256:83a1d2f1091078fb3f65876753b5302c5cfb6a41de64b9587b74cefa75157148",
                "sha256:8a21885513e6d915fe33a8ee5fdfa675433b61405ba13e2a69e62ee36828d7e2"
            ],
            "version": "==1.1.0"
        },
        "xlwt": {
            "hashes": [
                "sha256:a082260524678ba48a297d922cc385f58278b8aa68741596a87de01a9c628b2e",
                "sha256:c59912717a9b28f1a3c2a98fd60741014b06b043936dcecbc113eaaada156c88"
            ],
            "version": "==1.3.0"
        }
    },
    "develop": {
        "argparse": {
            "hashes": [
                "sha256:c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314",
                "sha256:62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4"
            ],
            "version": "==1.4.0"
        },
        "linecache2": {
            "hashes": [
                "sha256:e78be9c0a0dfcbac712fe04fbf92b96cddae80b1b842f24248214c8496f006ef",
                "sha256:4b26ff4e7110db76eeb6f5a7b64a82623839d595c2038eeda662f2a2db78e97c"
            ],
            "version": "==1.0.0"
        },
        "nose": {
            "hashes": [
                "sha256:dadcddc0aefbf99eea214e0f1232b94f2fa9bd98fa8353711dacb112bfcbbb2a",
                "sha256:9ff7c6cc443f8c51994b34a667bbcf45afd6d945be7477b52e97516fd17c53ac",
                "sha256:f1bffef9cbc82628f6e7d7b40d7e255aefaa1adb6a1b1d26c69a8b79e6208a98"
            ],
            "version": "==1.3.7"
        },
        "six": {
            "hashes": [
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
            ],
            "version": "==1.11.0"
        },
        "traceback2": {
            "hashes": [
                "sha256:8253cebec4b19094d67cc5ed5af99bf1dba1285292226e98a31929f87a5d6b23",
                "sha256:05acc67a09980c2ecfedd3423f7ae0104839eccb55fc645773e1caa0951c3030"
            ],
            "version": "==1.4.0"
        },
        "unittest2": {
            "hashes": [
                "sha256:13f77d0875db6d9b435e1d4f41e74ad4cc2eb6e1d5c824996092b3430f088bb8",
                "sha256:22882a0e418c284e1f718a822b3b022944d53d2d908e1690b319a9d3eb2c0579"
            ],
            "markers": "python_version < '2.7.9' or (python_version >= '3.0' and python_version < '3.4')",
            "version": "==1.1.0"
        }
    }
}

This example was generated with $ pipenv lock.

Pipfile.lock is always to be generated and is not to be modified or constructed by a user.

Do note how the versions of each dependency are recursively frozen and a hash gets computed so that you can take advantage of new pip security features.

Hashes are optional, because they can cause problems when using the same lockfile across different Python versions (e.g. a package will have different hashes according to different Pythons).

Pip Integration (eventual)

pip will grow a new command line option, -p / --pipfile to install the versions as specified in a Pipfile, similar to its existing -r / --requirement argument for installing requirements.txt files.

Install packages from Pipfile:

$ pip install -p
! Warning: Pipfile.lock (48d35f) is out of date. Updating to (73d81f).
Installing packages from requirements.piplock...
[installation output]

To manually update the Pipfile.lock:

$ pip freeze -p different_pipfile
different_pipfile.lock (73d81f) written to disk.

Notes:

# -p accepts a path argument, which defaults to 'Pipfile'.
# Pipfile.lock will be written automatically during `install -p` if it does not exist.

Ideas:

- Recursively look for `Pipfile` in parent directories (limit 3/4?) when ``-p`` is bare.

Useful Links

• Pipenv: Current reference implementation
• pypa/pip#1795: Requirements 2.0
• Basic Concept Gist (fork of @dstufft's)

Inspirations

• nvie/pip-tools: A set of tools to keep your pinned Python dependencies fresh.
• A Better Pip Workflow by Kenneth Reitz
• Lessons learned from Composer, Cargo, Yarn, NPM, Bundler and all Languages Owners at Heroku.

Documentation

The documentation for this project will, eventually, reside at pypi.org.

Discussion

If you run into bugs, you can file them in our issue tracker. You can also join #pypa on Freenode to ask questions or get involved.

Code of Conduct

Everyone interacting in the pipfile project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the PSF Code of Conduct.

pipfile's Issues

The difference between setup.py (pyproject.toml) and requirements.txt (Pipfile)

Why there even needs to be one? In PHP (Composer) and JS (npm, yarn), there is only one file type and libraries and applications work OK with these — libraries just don't need to commit their lockfile.

There is a difference between an application and a library, but on the pip's end, it's just a bunch of dependencies with versions specified.

Format of Pipfile

#10 organically grew into a discussion on which format should be used in Pipfile. Since that is a completely different question from whether Pipfile should be executable, I think that it deserves a dedicated issue.

You can obviously see comments there for the (brief) history on this.

find's max_depth off by one?

find with max_depth=1 will fail even if pipfile is in the current directory. Is that intentional?

Outdated README example?

In the README, the develop and default are both JSON arrays, but in the Pipfile.lock file in the examples folder, they are objects. e.g.

README:

    "development": [
        {"name": "nose", "version": "1.3.7", "hash": "..."},
    ]

examples/Pipfile.lock:

    "develop": {
        "nose": {
            "version": "==1.3.7",
            "hash": "sha256:dadcddc0aefbf99eea214e0f1232b94f2fa9bd98fa8353711dacb112bfcbbb2a"
        }
    }

Should it be assumed that the one examples/Pipfile.lock is the current spec since that's the generated format?

Also one is development and the other is develop.

Consider support for file like objects instead of filenames

The API is currently designed in a way that it tightly coupled with the file system. The parse function for example expects the Pipfile to be a filename: https://github.com/pypa/pipfile/blob/master/pipfile/api.py#L70

It'd great if we could find a way to decouple the API from the filesystem, at least in parts. This won't work for functions like walk_up, but when I look at parse it'd relatively easy to just pass in a file like object:

def parse(self, file_handle):
    # Open the Pipfile.
    content = f.read()
    ...

Same goes for the find function. Instead of calling os.getcwd explicitly in find, call it in walk_up:

 @staticmethod
def find(max_depth=3):
     """Returns the path of a Pipfile in parent directories."""
     i = 0
     for c, d, f in walk_up():
     ...

def walk_up(bottom):
    """mimic os.walk, but walk 'up' instead of down the directory tree.
    From: https://gist.github.com/zdavkeos/1098474
    """

    bottom = os.path.realpath(os.getcwd())
    ...

This would help external tools a lot to work with the API. There are probably still a lot of design decisions to make here, but if there's any interest in supporting external tools I'd be happy to submit a PR that addresses some of the points I mentioned earlier.

Rename Pipfile to Pipfile.toml

There is no benefit in not including the extension (a sense of mystery?).
In fact you will only have things to gain (editor highlighting, etc).

ref: Initially opened here.

Clarify that a Pipfile is a tool to generate Pipfle.lock, and not a build description on its own.

Executable build descriptions are a bad idea. See https://www.reddit.com/r/Python/comments/5e2vci/pipfile_a_new_and_much_better_way_to_declare/da9c2ku/ or uncountable blog posts and articles all over the Internet for very good examples. It should be common knowledge by now, still the same errors are made again and again.

Pipfile has a nice hybrid approach in that it allows developers to generate a static Pipfile.lock from a dynamic description. Tools can parse the static Pipfile.lock and know exactly what to do, while developers can work with the more convenient Pipfile and can be as smart or lazy as they want to. This approach might actually work very well.

For this to work, and for others like me to accept this idea, I thing the following point should be stressed out more: Tools work with Pipfile.lock exclusively. A build system should never automatically execute a Pipfile to generate a missing or outdated Pipfile.lock. The pipfile module should never be a build requirement.

Context managers

I think using context managers for package grouping looks more elegant.

Also I like the idea of usingsource as context manager from @dstufft's original proposal.

Lockfile doesn't include sub-dependencies

Originally reported by @charettes:

Shouldn't implicit dependencies be included in the lock files as well à la pip freeze?

Reasoning for that is simple: imagine package A depends on B == 1.0, and package B depends on C >= 1.0. Then C 1.1 comes out with some subtle changes that break B completely, and — BAM! — your build is broken.

The problem with this is, PyPI doesn't return packages dependencies (I've tried several APIs, no luck), so in order to lock dependencies we need t actually install them first. Doesn't seem like a big deal to me, though.

Function names in files

First off, I'd like to say THIS IS GREAT!

Im really excited about having a better version of requirements.txt. I wasn't sure where the best place for suggestions on this project are, so I figured an issue at this early stage was a good place. As always this is just my initial thoughts, do what you will with them (that includes closing this issue).

1. The dist() name doesn't make a whole lot of sense to me. I suspect its supposed to be short for distribution. My suggestion would be to make it about what the user wants to do. ie install, depends, require . These are all names that are in similar use in other package managers and I think make more sense. But names are also hard.
2. I know a lot of confusion comes from what setup.py is for and what requirements.txt is for. It would probably be really helpful to have some guidance on that, and where this fits in there.
3. I heard cargo from rust as a nifty feature called features. It allows a package to declare features that you could enable and you would get a set of transitive dependencies. So Django could have a postgres feature that you could opt into. When you do the psycopg2 package would get installed. If you don't then you only get sqlite. This allows the Django package owner to specify which version of postgres is installed while not making the user install the world. This would probably require a bunch of changes to setup.py/everything-else, but having a list of features that you could enable in this API might pave the way for that.

Anyway, really looking forward to using this!

Thanks for listening

Mark

Dealing with dependency conflicts

As we know, pip installs packages globally, unlike npm. Moreover, currently pip does not even try to deal with dependency conflicts (when you need both packages A and B, and they both depend on different versions of package C) in any way. It doesn't even notify you much about the fact it has just broken your dependencies.

Classic example, taken from here:
We develop our project or library and we need splinter and huxley as the dependencies. So we gonna install them:
$ pip install splinter
Pip installs splinter==0.7.5 and selenium==3.0.2 as splinter's dependency
$ pip install huxley

Collecting huxley
...
  Found existing installation: selenium 3.0.2
    Uninstalling selenium-3.0.2:
      Successfully uninstalled selenium-3.0.2
Successfully installed huxley-0.5 selenium-2.35.0

So we likely broke splinter, and, moreover, Pip said that everything was successfull. It even trolled us by saying successfully uninstalled package-that-we-use. And the user probably didn't notice anything, we don't read the logs line by line, when script ended successfully. It is bad.

Is it planned to define within this spec a behavior different from described above, when the packages with incompatible dependencies will be brought together in single Pipfile?

Use Pipfile.lock instead of Pipfile.freeze

lock is most popular filename suffix for storing exact package versions. Maybe we should use Pipfile.lock filename instead of Pipfile.freeze? Am I missing something?

Possible usecase for multiple groups.

Hi. I remember that we agree to do only default and dev group at this moment.

I want to explain my seeing of this feature if you decide to revisit this issue in the future.

We have customer who runs our web application written in python in its own cloud. Our application depends on few closed source packages developed for our customer by another team. We don't have access to this packages, but know provided api.

For testing purpose this team create stubs packages which we can install from PyPI installation running by our customer.

Before deploying new app release to staging infrastructure new code is reviewed by customer programmers team. In staging environment customer devops install real packages with known names from another PyPI instance running at customer staging environment. After QA engineers approved new release same installation shipped to the production environment where customer PyPI instance running on another url.

So it's good to have something that can handle this tricky setup easily. I see it like

[packages]
customer_package_foo = "*"
customer_package_bar = "*"

[packages:staging]
<<inherit packages from previous section>>
<<pin real dependencies of foo and bar packages>>
<<provide staging pypi url>>

[package:prod]
<<inherit packages from staging>>
<<provide prod pypi url>>

Dependencies by python version

It should be possible to set dependencies by Python version to allow installing backported modules only when needed.

installing packages from git broken

Fresh install:

$ pipenv --version
pipenv, version 3.3.6

Simple Pipfile:

$ cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
crayons = { git="https://github.com/kennethreitz/crayons.git", ref="v0.1.2" }

Create lock file:

$ pipenv lock
Creating a virtualenv for this project...
[...]
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock!

... which generates this:

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "39b2447b7dd6976804de490321577a2db0f4fc4327e0795a93436ea69afad9a8"
        },
        "requires": {},
        "sources": [
            {
                "verify_ssl": true,
                "url": "https://pypi.python.org/simple"
            }
        ]
    },
    "default": {
        "crayons from git+https://github.com/kennethreitz/[email protected]#egg=crayons": {
            "hash": "sha256:a292b7ef29131345fd329ff5a94d1e2e59569be4dbd9fa8b99ff7114d72102a6",
            "version": "==0.1.2"
        },
        "colorama": {
            "hash": "sha256:a4c0f5bc358a62849653471e309dcc991223cf86abafbec17cd8f41327279e89",
            "version": "==0.3.7"
        }
    },
    "develop": {}
}

And now installing breaks:

$ pipenv install
No package provided, installing all dependencies.
Pipfile found at /[...]/Pipfile. Considering this to be the project home.
Installing dependencies from Pipfile.lock...
An error occured while installing!
Invalid requirement: 'crayons from git+https://github.com/kennethreitz/[email protected]#egg=crayons==0.1.2'
It looks like a path. Does it exist ?


To activate this project's virtualenv, run the following:
 $ pipenv shell

These are the versions in my env:

$ pip freeze
appdirs==1.4.0
blindspin==2.0.0
click==6.7
click-completion==0.2.1
colorama==0.3.7
crayons==0.1.2
delegator.py==0.0.8
Jinja2==2.9.5
MarkupSafe==0.23
packaging==16.8
parse==1.6.6
pew==0.1.26
pexpect==4.2.1
pipenv==3.3.6
pipfile==0.0.1
ptyprocess==0.5.1
pyparsing==2.1.10
pythonz-bd==1.11.4
requests==2.13.0
requirements-parser==0.1.0
resumable-urlretrieve==0.1.4
six==1.10.0
toml==0.9.2
virtualenv==15.1.0
virtualenv-clone==0.2.6

Is this project considers to be an official replacement for pip's requirements.txt?

Hello! Sorry for asking such a question as an issue, but search did not helped me about official position of this project.

Is this project considers to be an official replacement for pip's requirements.txt?

Let's use TOML

Advantages:

1. --save, --save-dev, --save-exact could be implemented without much effort

2. common file format, so we could use existing editor capabilities

3. In the future we possible could converge existing pyproject.toml, setup.py and this file, into one.

This is a little specs, I've created, maybe it will be of any use (it's not complete, VCS has to be expanded, maybe I've forgotten some other cases)

[dependencies]
# plain, command: pip install --save Django
django = ">=1.10.1"

# command: pip install --save-exact Django 
django = "==1.10.1"

# command: pip install --save Django==1.10.1 
# Any specified specifier goes exactly (even without --save-exact)
django2 = "==1.10.1"

# full version
SomeProject = {version = "==5.4"} # denotes what version exists as a part of package metadata

# environment markers
SomeProject2 = {version = "==5.4", markers = {python_version = "< 2.7", sys_platform = "win32"}}
# OR
[dependencies.SomeProject2.markers]
sys_version = "< 2.7"
sys_platform = "win32"

# particular file
# all non-obvious dependencies should be specified as a table (not inline), it's more verbose, which is a good thing here
[dependencies.numpy] # appears after --save
file = "./downloads/numpy-1.9.2-cp34-none-win32.whl"

# OR 

numpy = {file = "./downloads/numpy-1.9.2-cp34-none-win32.whl"}
wxpython = {file = "http://wxpython.org/Phoenix/snapshot-builds/wxPython_Phoenix-3.0.3.dev1820+49a8884-cp34-none-win_amd64.whl"}

# per-requirement overrides
#FooProject >= 1.2 --global-option="--no-user-cfg" \
#                  --install-option="--prefix='/usr/local'" \
#                  --install-option="--no-compile"
# maps to 
FooProject = {version = ">=1.2", setup_py_options = """--global-option="--no-user-cfg"
                                                    --install-option="--prefix='/usr/local'"
                                                    --install-option="--no-compile"""} # multiline string, only multiline that in valid in TOML

# OR
# if used in conjunction with --save (it's automatically generated, that's why doesn't really matter if it's verbose or not)
[dependencies.FooProject] 
version = ">=1.2"
setup_py_options = """--global-option="--no-user-cfg"
                    --install-option="--prefix='/usr/local'"
                    --install-option="--no-compile"""

# if necessary to add hashes
# https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
hashes = """ # maybe convert to array
--hash=sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 \
--hash=sha256:486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7
"""

# VCS 
flask = {git = "git repo", revision = "hash"}

# extras (dev, test, production), dev could be special-cased as [dev_dependencies]
[extras.PDF]
ReportLab = ">=1.2"

No matching distribution found for pipfile

print 'No matching distribution found for pipfile' when use pip install pipfile.
My tools version as below:
pip version: 9.0.1
python version: 2.7

Pip version in the Pipfile.lock

Bundler is an amazing tool in ruby-land.
I hope you guys study some of it, as it works really really well.
Specifically, A major pain point there was an addition of the specific "Bundler" version to the Gemfile.lock .

This was a painful migration, but later on allowed much more power to the Bundler authors, if it's in upgrades or in specific lock file changes.
I think if you add this and the specific bundler logic (only update when newer version exists) it will save much pain later on -
rubygems/bundler#3697

Consider indicating sub-dependencies in pipfile package list

pipdeptree style

pipdeptree==0.10.1
  - pip [required: >=6.0.0, installed: 9.0.1]
Sphinx==1.6.2
  - alabaster [required: >=0.7,<0.8, installed: 0.7.10]
  - babel [required: !=2.0,>=1.3, installed: 2.4.0]
    - pytz [required: >=0a, installed: 2017.2]
  - docutils [required: >=0.11, installed: 0.13.1]
  - imagesize [required: Any, installed: 0.7.1]
  - Jinja2 [required: >=2.3, installed: 2.9.6]
    - MarkupSafe [required: >=0.23, installed: 1.0]
  - Pygments [required: >=2.0, installed: 2.2.0]
  - requests [required: >=2.0.0, installed: 2.17.3]
    - certifi [required: >=2017.4.17, installed: 2017.4.17]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.3]
    - idna [required: <2.6,>=2.5, installed: 2.5]
    - urllib3 [required: >=1.21.1,<1.22, installed: 1.21.1]
  - setuptools [required: Any, installed: 36.0.1]
  - six [required: >=1.5, installed: 1.10.0]
  - snowballstemmer [required: >=1.1, installed: 1.2.1]
  - sphinxcontrib-websupport [required: Any, installed: 1.0.1]

is more friendly and explicit than

alabaster = "==0.7.10"
Babel = "==2.4.0"
certifi = "==2017.4.17"
chardet = "==3.0.3"
docutils = "==0.13.1"
idna = "==2.5"
imagesize = "==0.7.1"
Jinja2 = "==2.9.6"
MarkupSafe = "==1.0"
pipdeptree = "==0.10.1"
Pygments = "==2.2.0"
pytz = "==2017.2"
requests = "==2.17.3"
snowballstemmer = "==1.2.1"
Sphinx = "==1.6.2"
sphinxcontrib-websupport = "==1.0.1"
urllib3 = "==1.21.1"

I'm not saying copy the syntax, just the idea of showing which package installed / is dependent upon which package. It is nice.

https://pipfile.pypa.io/ doesn't work

https://pipfile.pypa.io/ give me a DNS address could not be found error.

Why not use config.cfg ?

We already have so many files, a manifest, requirements files, config.cfg, tox, etc.

Let's remove the need for them and just have new sections in setup.cfg.

Custom Pipfile naming

Is there a way to give a custom name for Pipfiles?

See here for context:
https://github.com/kennethreitz/pipenv/issues/248

Arbitrary group definitions with new toml file format.

If I understand correctly previous version of Pipfile was build around python like syntax and allows define arbitrary groups like that:

package('django')

with group('develop'):
    package('django-debug-toolbar')

with group('test'):
    package('pytest')

with group('develop', 'test'):
    package('django-extensions')

And I could define any group I want and install it explicitly, correct?

How can I define similar pipfile with recent toml syntax?

Address likely FAQs about Pipfile in the README

I know this project was just open-sourced and is still under active development, so consider this a reminder to update the README with some additional info when the time is right.

Some questions that immediately popped into my mind, which others are likely to ask, include:

1. How does Pipfile relate to setup.py and the upcoming pyproject.toml? Is it the same relationship as the one described here between requirements.txt and setup.py?
2. With the planned setup.py to pyproject.toml transition, we seem to be moving away from using Python and more towards using a declarative syntax for specifying build system requirements. With this proposed requirements.txt to Pipfile transition, however, we seem to be moving in the opposite direction. Could you comment on this?
3. Does Pipfile just use a Python-like syntax, or will it actually run Python code, like setup.py?
4. What is the purpose of Pipfile.lock, and will packaging end-users need to care about this file?

Add usage to readme

I've been looking for about 15 minutes now after discovering a Pipfile in a project I was looking at. I can't figure out how to use it. Maybe you could add a simple example usage to your readme?

--save flag while installing new packages?

In the npm world, you can install new packages like so:

npm install --save <package>

and the --save flag will write the package details to the package.json (pipfile equivalent). I feel this will be a neat inclusion for pip/pipfile

[request] Another release on pypi

Could we please consider releasing another version on pypi, which incorporates changes allowing for it to work better in Windows environment?

This would greatly help in allowing basic functionality in pipenv on Windows.

consider: please change the names of these files

Rather than naming it "Pipfile.lock", could we call these something else before they catch on?

1. File extensions are a good idea; Filefiles are an antipattern. This new file should be <something>.pipfile. Lots of software (the mac's Finder, windows's Explorer, Nautilus, Vim, Emacs, Apache, Nginx, anything that knows how to read /etc/mime.types) can infer a MIME type from an extension easily and only from a full filename with great difficulty if at all. Even if it's always called Pipfile.pipfile, please consider a unique file extension.
2. A file whose name ends in .lock is a file that you call flock() on, or perhaps, over NFS, that you call open(..., O_CREAT | O_EXCL) on. In other words, it is a lock. I realize that this naming convention is sort of by extension to Ruby's Gemfile.lock, but (A) the contents of the file are of a different MIME type than Gemfile.lock so it should be a different extension, and (B) Gemfile.lock is one of the worst, most baffling names ever given to an abstraction. Other suggestions: KnownGood.pipfile, ExactVersions.pipfile, Complete.pipfile, Exhaustive.pipfile, Repeatable.pipfile...?

Don't get me wrong - this is extremely useful functionality and I'm looking forward to using it! I just really hope that we can avoid repeating a templated set of mistakes that many programming tools seem intent on promoting these days.

Consider scripts section similar to npm package.json

Hi, thanks for yet another awesome tool!

I wander if there is any plans to add commands abbreviations for example in scripts section.

So instead of typing each time pipenv run python -m unittest discover it will be enough to run pipenv test (or maybe pipenv apply test) if we have this in the Pipfile:

[[scripts]]
test = "python -m unittest discover"

Moved from https://github.com/kennethreitz/pipenv/issues/153

Please consider lowercase: pipfile and pipfile.lock

Is there a good reason why Pipfile{,.lock} couldn't be all lowercase? It sucks to have to hit the SHIFT-key when using the commandline, IMHO. At least, please consider accepting both pipfile and Pipfile.

Take inspiration from Yarn, Bundler and Cargo

Perhaps you've already done this, but if not I'd suggest looking at how similar systems has solved issues around deterministic builds, dependency inconsistencies, the lock-file and similar. A non-exhaustive list:

• Bundler (ruby)
• Cargo (rust)
• Yarn (node)
• Composer (php)
• CocoaPods (objective-c/swift)

If you have insights or tidbits from these projects, feel free to add them in the comments below.

[question] How to integrate with setup.py?

Using requirements.txt I can do it:

from pip.req import parse_requirements
requirements = [str(r.req) for r in
                parse_requirements('requirements.txt', session=False)]
test_requirements = [str(r.req) for r in
                     parse_requirements('requirements-test.txt', session=False)]

How can I do the same using Pipfile?

specifying relative path as source for package

I've seen the example for using a git repository as an alternative source:

pinax = { git = 'git://github.com/pinax/pinax.git', ref = '1.4', editable = true }

What would the -e . in requirements.txt equivalent be?

Executable File Format, Yes or No?

There have been a number of issues/comments that tend to trace back to a single question.. Should Pipfile be executable or not?

The relevant comments are:

@defnull said in #8 (comment)
Parsing and editing Pipfile on the other hand is significantly more complicated that requirements.txt. IDEs will most likely not support Pipfile editing for a long time.

@jayfk said in #8 (comment)
On top of that, please don't forget server side tools working with dependencies (pyup.io, requires.io, etc.). There's no way to support Pipfiles if they allow to run arbitrary Python code. Local developer tools might have a chance to establish a working solution over time, server side tools don't.

@defnull said in #9 (comment)
Build or install tools (pip) cannot check if Pipfile and Pipfile.lock are out of sync without executing the Pipfile (which is bad, see #7) and cannot warn the user.

@defnull said in #7 (comment)
Executable build descriptions are a bad idea. See https://www.reddit.com/r/Python/comments/5e2vci/pipfile_a_new_and_much_better_way_to_declare/da9c2ku/ or uncountable blog posts and articles all over the Internet for very good examples. It should be common knowledge by now, still the same errors are made again and again.

Pipfile has a nice hybrid approach in that it allows developers to generate a static Pipfile.lock from a dynamic description. Tools can parse the static Pipfile.lock and know exactly what to do, while developers can work with the more convenient Pipfile and can be as smart or lazy as they want to. This approach might actually work very well.
For this to work, and for others like me to accept this idea, I thing the following point should be stressed out more: Tools work with Pipfile.lock exclusively. A build system should never automatically execute a Pipfile to generate a missing or outdated Pipfile.lock. The pipfile module should never be a build requirement.

@takluyver said in #6 (comment)
I don't like specifying metadata in executable files in general. #8 gives one reason why not - it's very hard to reliably modify scripts programmatically.

There's also some reaction on https://www.reddit.com/r/Python/comments/5e2vci/pipfile_a_new_and_much_better_way_to_declare/

Double requirements given

Hello! When using pipenv i have an pipfile error: Double requirements given.

You can reproduce it with this Pipfile using a command pipenv install.

I am not sure if it is pipenv or Pipfile's problem, so i decided to submit and issue here. As far as i know, many users are facing this problem.

Include a 'version' tag in Pipfile.lock

How to deploy ranged version for library

Hello.

I differential python library to "application". A library ends up typically on pypi to be used by another application. One should never do a pip install thislibrary in production. Should comes from a pip install thisapplication that consequently install other apps and libraries.

For me, application should indeed have frozen dependencies, to ensure perfect reproduction. But libraries deployed on pypi should better be using "range" for acceptable version. For exemple, if two libraries depends on frozen version of another lib (say 1.0 and 2.0), there will be a conflict. Libraries dependencies are better described with range (I know my lib will work with this dependencies from version 1.0 to 3.0, or require a version >1.1, and so on).

Is it possible to do this with the pipfile and how? It is possible to describe this subtility in the documentation? "How to handle dependencies of a library or an application" ?

Thanks

Non-python dependencies

Does the pipfile manage the required C libraries and platform specific packages too?

Risk of outdated Pipfile.lock after editing Pipfile.

In addition #8, this is perhaps the most serious usability problem with the current approach: For a developer it is very easy to forget about Pipfile.lock after editing Pipfile, especially because he is not supposed to care about the lock file.

This is not an issue with pure approaches (setup.py or requirements.txt) because there is only one version of truth. Pipfile introduces an intermediate step that translates one truth into another, and different tools look at different files. This just asks for trouble. I can already see the broken builds and "fix: Forgot to build Pipfile" commits waiting to happen.

A very common workflow is to install a dependency manually, check if it works, and then add it to requirements.txt or just call pip freeze again. With Pidfiles, there is no pip freeze (see #8) and you have to invoke a command-line utility after editing the Pidfile to make things happen. Unfortunately, you just manually installed the dependency, so everything works anyway (on your computer) and there is no indication that your Pidfile.lock is out of sync if you forget this step. You are a single git commit -a away from a broken build.

Build or install tools (pip) cannot check if Pipfile and Pipfile.lock are out of sync without executing the Pipfile (which is bad, see #7) and cannot warn the user.

How should Extra-index-url be treated in Pipfile?

Will it be just another entry in [[source]] with a boolean extra=True?
Or pipfile will consider extra-index-url to be just like any other source? if so, anything that uses Pipfile will have to do some try/excepts to see if the current package is present in each of the listed sources (because some of them can be a extra, and contain just some specific listed packages)

Default `pip install` behavior?

Currently, pip install outputs You must give at least one requirement to install (see "pip help install"). It would be nice to be able to do just pip install in a directory with a Pipfile.lock and have it recognize and run the installation automatically based on that file. This behavior is identical to pip install -p ("when -p is bare") as described in the README but provides a default for pip install without needing the -p flag, like npm install. If no Pipfile.lock file is found, pip install can output its current error message.

Additionally, perhaps pip install can recognize a Pipfile if there is no Pipfile.lock file, run the install, and generate a corresponding Pipfile.lock file.

Thoughts?

Discouraging overly complex Pipfile definitions

One of the concerns raised in previous discussions of this idea is that it risks inheriting all the problems that have historically plagued setup.py, such as having to execute arbitrary code just to extract project metadata.

Having Pipfile.lock mitigates a large part of that problem (since the generated JSON metadata will already be static at install time), and a gemspec-style pyproject() directive would make it straightforward to delegate to a library-style pyproject.toml file when a given repo supports both direct deployment as a service and publication as a versioned component.

However, it would also be possible to explicitly discourage (but not entirely prevent) the inclusion of arbitrary control flow logic by blocking access to most of the builtins when executing Pipfile code. For example:

>>> limited_namespace = dict(__builtins__={})
>>> exec("import sys", limited_namespace, {})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
ImportError: __import__ not found

This wouldn't provide a security boundary (sandboxing Python code isn't that simple), but it would mean that imports and the open() command wouldn't work by default, and folks would have to actively work to gain access to the standard builtins in order to change that.

Support for requiring packages at install time

Is there support or planned support for installing packages before installation the current package (at build time), such as Cython or NumPy, as described in https://www.python.org/dev/peps/pep-0518/

Idea: using YAML instead of TOML

Kind of like shards does. It's quite a bit more concise than TOML and even looks kind of Python-y.

I know the answer to this is probably going to be "big change, not enough gain", "conciseness isn't a good reason to switch", "harder to parse", etc...but I figured this still is worth a shot. ;)

What abot a requirements.txt to pipfile convertor?

Is there any converter?

Or any unified schema for starting to develop it?

Idea: add hashes in generated .lock file

Since version 8, pip can now process requirements.txt files and check their hashes after download.

https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

I think it would be a good idea to store those hashes in the generated .lock file

(A big inconvenience af the hash-checking mode of pip is that you have to specify the hash for every package and their dependencies, recursively ...)

.freeze renamed to .lock?

I recall a long discussion on #43 and elsewhere, where a decent consensus was reached to change to freeze files, so as to avoid confusion with file locking. I may be confused, but this seems to have been unilaterally reverted? I can't find any further discussion.

It will be very hard for tools (IDEs) to change dependencies programmically.

Tools and IDEs cannot edit the Pipfile.lock directly, because changes will be lost once the Pipfile is executed again. Parsing and editing Pipfile on the other hand is significantly more complicated that requirements.txt. IDEs will most likely not support Pipfile editing for a long time.

To put it in perspective: Pipfile.lock is a read-only approach. Developers are forced to manually edit the Pipfile with little to no tooling support in order do change the dependency settings of their project. This is a step backward compared to requirements.txt or other static file approaches.

Could pipfile be designed to support a MetaPathFinder implementation that installs the requirements 'on-demand' ?

I learned about MetaPathFinder class from importlib and created this hack:

http://lonetwin.github.io/blog/html/2016/05/16/auto_install_missing_python_modules.html

Since I discovered this hack, I have been thinking that something of an 'install-on-missing-import' feature would be, in theory, a possibility,

I've been wrestling with getting an implementation which works with requirements.txt for and in fact, I do have a rough implementation which parses the requirements.txt and installs the requirements at import time. This works for most simple installations, although I haven't tested it extensively.

I would like to suggest that maybe pipfile could designed (and possibly provide a reference implementation) of a MetaPathFinder which ensures that the developers and users of modules don't even have to worry about running a command to install anything. Just executing or importing the module should install the necessary dependencies.

This is just a suggestion and I'm ok if people think this idea is ridiculous. That said, knowing the reasons why such a thing would be ridiculous would be appreciated.