The difference between setup.py (pyproject.toml) and requirements.txt (Pipfile) about pipfile

[Screwtapello]

I'm confused by this too, but I understand things a bit differently:

Old world:

• setup.py contains abstract dependencies
  • for libraries, these dependencies affect other packages that use the library
  • for applications, this is mostly used by developers working on new versions
• requirements.txt contains concrete dependencies
  • not used for libraries
  • used for repeatable deployment of applications

New world:

• setup.py contains abstract dependencies for libraries (so that they can be stored in wheels and uploaded to PyPI)
• Pipfile contains abstract dependencies for applications (used by developers working on new versions)
• Pipfile.lock contains concrete dependencies

I wish abstract dependencies could be in the same place, in the same (non-Turing-complete) format for both libraries and applications.

[k0nserv]

In contrast to PHP and JS Ruby/Bundler has

• .gemspec - Specification file for libraries (Equivalent to setup.py)
• Gemfile - Desired dependencies (Equivalent to Pipfile)
• Gemfile.lock - Locked dependencies, generated from Gemfile (Equivalent to Pipfile.lock)

For libraries that are consumed by others and published on the repository you would use gemspec for Ruby/Bundle and setup.pyfor Python. For projects (scripts, web backends, etc) you forego the .gemspec/setup.py and instead use Gemfile/Pipfile and Gemfile.lock/Pipfile.lock. Libraries don't commit their lock files an depend on semver for transitive dependencies that are resolved at install-time. I think Pipfile is leaning more in this direction than JS (where both a package specification for a library and dependencies for a non-library project live in the same file).

Cargo has both package and binary specifications in the same file similar to JS with package.json

• Cargo.toml (Gemfile, Pipfile, gemspec, setup.py)
• Cargo.lock (Gemfile.lock, Pipfile.lock)

CocoaPods also uses a this split with

• .podspec (.gemspec, setup.py, .crate)
• Podfile (Gemfile, Pipfile, Cargo.toml)
• Podfile.lock (Gemfile.lock, Pipfile.lock, Cargo.lock)

I'm not involved with this project, but I suspect this is roughly what the aim is

[notpushkin]

@nchammas Yes, I've read that. It doesn't explain why we need two separate formats, though.

[nchammas]

So you're saying you understand the different roles these two files play but you don't see why they can't be captured in the same file? I thought @dstufft's post explained why clearly:

So Why Does Abstract and Concrete Matter?

You’ve read this far and maybe you’ve said, ok I know that setup.py is designed for redistributable things and that requirements.txt is designed for non-redistributable things but I already have something that reads a requirements.txt and fills out my install_requires=[...] so why should I care?

This split between abstract and concrete is an important one. It was what allows the PyPI mirroring infrastructure to work. It is what allows a company to host their own private package index. It is even what enables you to fork a library to fix a bug or add a feature and use your own fork. Because an abstract dependency is a name and an optional version specifier you can install it from PyPI or from Crate.io, or from your own filesystem. You can fork a library, change the code, and as long as it has the right name and version specifier that library will happily go on using it.

A project can only have 1 set of abstract dependencies (setup.py/pyproject.toml), but different users working with that project can have different sets of concrete dependencies (requirements.txt/Pipfile) which allow them to fulfill those abstract dependencies from PyPI mirrors, private package indexes, personal forks on GitHub, or somewhere else.

So if project Car depends on Engine (an abstract dependency), I can choose to install Car but grab Engine specifically from a fork I made on GitHub (a concrete dependency) that has some performance improvements. Meanwhile, someone else working at a big company that doesn't want to depend on external services to build and deploy their internal Python projects can choose to install both Car and Engine from their private package index as opposed to PyPI (another concrete dependency).

You can't merge these two types of dependencies together into one file without hampering people's ability to choose where to get their dependencies from.

Does that clarify why setup.py/pyproject.toml and requirements.txt/Pipfile need to be different things?

[notpushkin]

@nchammas Thanks for the clarification! So basically, the workflow now is:

1. setup.py/pyproject.toml (abstract dependencies / version specs (like >= 1.8.0, < 2.0)) — a lot of these; one for each library we depend on
2. Pipfile (where to get them; may be different between forks)
3. Pipfile.lock (which exact versions to get).

Did I get it right?

What happens if setup.py gets updated? Pipfile gets out of date, we need to sync it; how do we do it?
Example: imagine the following situation:

# Car's Pipfile
with source("https://super-secret-cheeseshop.pythonlabs.com/", verify_ssl=True):
    package("Engine", ">=1.0.0")

# Engine's setup.py
setup(
    name="Engine",
    version="1.0.0",
    packages=["engine"],
)

At some point Engine's author decides to update it:

# Engine's new setup.py
setup(
    name="Engine",
    version="1.1.0",
    packages=["engine"],
    install_requires=["Cylinder >= 1.1"],
)

Now we do pip freeze -p Pipfile. Where do we get Cylinder from: the PyPI or the super secret cheeseshop?

[ncoghlan]

@askabelin If a project isn't meant to be importable, you don't need a setup.py at all, and can just use requirements.txt or a Pipfile. Yes, a setup.py can be used for the application-or-service use case, but the extra complexity isn't typically needed in such cases, and may give folks the misleading impression that the component is intended to be usable as a library.

[nchammas]

@iamale - The difference is explained here: https://caremad.io/posts/2013/07/setup-vs-requirement/

[Julian]

@nchammas pip (requirements.txts) does not currently support specifying where a specific package comes from.

It just lets you specify globally your set of indexes.

[askabelin]

May be I am missing something but this discussion doesn't answer the question "what can go wrong if you'll put your app's concrete dependencies in setup.py and you don't have abstact ones". Not a good practice. Abstract matters. It is still not enough for data duplication in my opinion.

This split between abstract and concrete is an important one. It was what allows the PyPI mirroring infrastructure to work. It is what allows a company to host their own private package index.

Why do we need "abstract" deps if app is not supposed to be imported in other projects and to be uploaded to public PyPI (because it is private app)? It is not a problem to deploy it from private PyPI as well.

A project can have only have 1 set of abstract dependencies (setup.py/pyproject.toml), but different users working with that project can have different sets of concrete dependencies.

Different sets of dependencies (dev/ops) can be resolved with extras_require. People can play with versions but versions that work for sure can be specified in setup.py. What is the purpose of abstract dependencies if application was not tested with it? Just "should work".

I agree that setup.py is a bad place for index url. Index is more related to infrastructure than to code. So it could be provided externally with Ansible / SALT etc.

May be someone has real use case where just setup.py was a problem. Please share it.

[FRidh]

libraries just don't need to commit their lockfile.

Indeed, they don't have to, since for building/installation setup.py is sufficient. However, as developer of the library you might want to share your environment for testing purposes. Pipfile.lock would provide a file format for that use (like requirements.txt now). And for when deploying an environment.

[ionrock]

To provide a really practical example, I have a project at work where we do a few things.

• setup.py gets necessary app dependencies
• requirements.txt is used for defining an internal pypi and adding runtime deps (db driver to use)
• dev_requirements.txt is used to install test/dev reqs like pytest, mock, webtest, etc.
• ops_requirements.txt is used for a parallel operational venv that became necessary b/c a lib conflicted with the app requirements!

This project is for some integration so its requirements might be complicated than others, but generally, keeping app requirements you need in setup.py and runtime or deployment requirements in requirements.txt ends up helpful for the simple reason you can use pip flags in requirements.txt.

[nchammas]

That's a good question. I'm not sure what the answer is, but we could probably find out by testing this scenario with requirements.txt, since Pipfile is filling the same role.

@dstufft / @ncoghlan: If I instruct pip to install a package from a specific source (e.g. a private index), does it automatically try to install all that package's dependencies from the same source?

Or would it grab just the specifically mentioned package from the private index, and everything else. including dependencies of that package, from the default source (PyPI)?

[pradyunsg]

Unless OP (or someone else) has any more questions regarding why there's a proposal to have 3 separate files in the Python Packaging System to manage dependencies, let's close this issue.

[dstufft]

I'm going to go ahead and close this. There doesn't seem to be anything actionable here. If I'm wrong please speak up or open another issue, thanks.

[ryanhiebert]

@k0nserv : You mentioned there being a .crate file in Cargo that's roughly equivalent to setup.py / pyproject.toml, .gemspec, .podspec, etc, but that doesn't line up with what I know or what I can find. All I can find about .crate is the files that are uploaded to crates.io (like .whl files in Python, IIUC).

Can you help clear up my confusion?

If it's true that there's no equivalent to those files in Cargo, that seems like a signal that we may wish to analyze. Cargo's constraints are different enough from ours that there might be some reason that we need the distinction, but if the designers of Cargo, having already lived with the design of .gemspec, Gemfile, and Gemfile.lock, decided to not emulate it, then that may be a split that we ultimately may not want to make either.

---

This article on the gem files, as well as this comment from @dstufft about the utility of the split, makes sense to me. I just wonder if there's something that the folks that made Cargo know that we don't, and what they'd recommend to us.

http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/
jazzband/pip-tools#418 (comment)

[k0nserv]

@ryanhiebert Yes you are correct. Rust and Cargo uses Cargo.toml to specify both libraries and binaries. However as in other languages when building a library to be consumed by others you don't include Cargo.lock in the project instead relying on transitive dependencies being installed when the library itself is installed.

[dstufft]

One of the folks that made Cargo is the person who wrote that article :) One of the main differences though is that there isn't really a runtime declaration with Cargo. Everything is build time and Cargo produces a static binary. That changes the math a bit.

[ryanhiebert]

Yeah, I knew that he was involved with Cargo, which seems like it may be evidence that we need to consider that choice, since Cargo was made well after that post was written. Since he knew the benefits and pitfalls of using it in Ruby at the time, his choice to not replicate it for Rust means something to me.

It makes sense to me that the compilation nature of Rust could change the math, but I haven't come up with how it changes it. It seems like virtually all of the reasoning that we have behind doing the three-file approach still applies, even to Rust.

My own reasoning at the moment really wonders if having Cargo.toml fill both the slot of pypackage.toml and Pipfile is actually a bad choice. In my own limited work with Rust, I've created a library that included a reference implementation binary that used that library. The library shouldn't have the lock file checked in, but the binary reasonably should. So I had to choose, and had to go with keeping the lock file out of the repository so I didn't screw up the usage of it as a library. I could make them separate repositories, but that didn't seem worthwhile.

Thus, my own thinking makes me think that the split is a good thing. I wonder if we could get @wycats to comment on the factors that lead to the decision that Cargo made.

Perhaps for Cargo several of the things that are constraints for us were deemed unnecessary. If enough of our reasons for keeping them separate were deemed of enough less utility for Cargo, then the overhead of an extra file may have been enough to decide to take the bad with the good, and skip the added file.

[ncoghlan]

Since it's much easier to add features than it is to remove them, I'd advise any new packaging system to start with a single file the way Rust has, and then use experience-driven-pain to decide whether or not they need anything more complex than that.

For the specific case you gave, it seems that Rust has first class "example application" support built (http://doc.crates.io/manifest.html#examples) and otherwise requires that applications and libraries by laid out differently (http://doc.crates.io/manifest.html#the-project-layout). It also has workspaces to more explicitly manage scenarios where projects are developed and published as multiple crates in a single repository: http://doc.crates.io/manifest.html#the-workspace-section

The big difference in Python is that we have two rather distinct problems to solve:

• how to provide an opinionated and coherent publisher-centric dependency management experience for folks starting a new project or actively changing the workflow for an existing one (i.e. pipenv)
• how to provide a consumer-centric "minimum required build bootstrapping capability" that can be readily incorporated into existing build and deployment tools (i.e. pyproject.toml)

As a new ecosystem, Rust had the luxury of imposing Cargo.toml and Cargo.lock as their publishing standard from the start, rather than Python's more complicated history of retrofitting first distutils (~1998), then setuptools (~2004), then virtualenv (~2007), then pip (~2011), and now pyproject.toml and pipenv (~2017).

[ryanhiebert]

Since it's much easier to add features than it is to remove them

I think that's the motivation I was looking for. Makes perfect sense to try a more limited approach first, then expand if necessary. Thank you. Clearly, Python is way past that already.

[taion]

I followed up with a concrete proposal in #98 that I believe gives us something actionable and pragmatic that could improve DX for maintaining Python libraries.