Comments (21)
I'm confused by this too, but I understand things a bit differently:
Old world:
setup.py
contains abstract dependencies- for libraries, these dependencies affect other packages that use the library
- for applications, this is mostly used by developers working on new versions
requirements.txt
contains concrete dependencies- not used for libraries
- used for repeatable deployment of applications
New world:
setup.py
contains abstract dependencies for libraries (so that they can be stored in wheels and uploaded to PyPI)Pipfile
contains abstract dependencies for applications (used by developers working on new versions)Pipfile.lock
contains concrete dependencies
I wish abstract dependencies could be in the same place, in the same (non-Turing-complete) format for both libraries and applications.
from pipfile.
In contrast to PHP and JS Ruby/Bundler has
.gemspec
- Specification file for libraries (Equivalent tosetup.py
)Gemfile
- Desired dependencies (Equivalent toPipfile
)Gemfile.lock
- Locked dependencies, generated fromGemfile
(Equivalent toPipfile.lock
)
For libraries that are consumed by others and published on the repository you would use gemspec
for Ruby/Bundle and setup.py
for Python. For projects (scripts, web backends, etc) you forego the .gemspec
/setup.py
and instead use Gemfile
/Pipfile
and Gemfile.lock
/Pipfile.lock
. Libraries don't commit their lock files an depend on semver for transitive dependencies that are resolved at install-time. I think Pipfile is leaning more in this direction than JS (where both a package specification for a library and dependencies for a non-library project live in the same file).
Cargo has both package and binary specifications in the same file similar to JS with package.json
Cargo.toml
(Gemfile
,Pipfile
,gemspec
,setup.py
)Cargo.lock
(Gemfile.lock
,Pipfile.lock
)
CocoaPods also uses a this split with
.podspec
(.gemspec
,setup.py
,.crate
)Podfile
(Gemfile
,Pipfile
,Cargo.toml
)Podfile.lock
(Gemfile.lock
,Pipfile.lock
,Cargo.lock
)
I'm not involved with this project, but I suspect this is roughly what the aim is
from pipfile.
@nchammas Yes, I've read that. It doesn't explain why we need two separate formats, though.
from pipfile.
So you're saying you understand the different roles these two files play but you don't see why they can't be captured in the same file? I thought @dstufft's post explained why clearly:
So Why Does Abstract and Concrete Matter?
You’ve read this far and maybe you’ve said, ok I know that setup.py is designed for redistributable things and that requirements.txt is designed for non-redistributable things but I already have something that reads a requirements.txt and fills out my install_requires=[...] so why should I care?
This split between abstract and concrete is an important one. It was what allows the PyPI mirroring infrastructure to work. It is what allows a company to host their own private package index. It is even what enables you to fork a library to fix a bug or add a feature and use your own fork. Because an abstract dependency is a name and an optional version specifier you can install it from PyPI or from Crate.io, or from your own filesystem. You can fork a library, change the code, and as long as it has the right name and version specifier that library will happily go on using it.
A project can only have 1 set of abstract dependencies (setup.py
/pyproject.toml
), but different users working with that project can have different sets of concrete dependencies (requirements.txt
/Pipfile
) which allow them to fulfill those abstract dependencies from PyPI mirrors, private package indexes, personal forks on GitHub, or somewhere else.
So if project Car
depends on Engine
(an abstract dependency), I can choose to install Car
but grab Engine
specifically from a fork I made on GitHub (a concrete dependency) that has some performance improvements. Meanwhile, someone else working at a big company that doesn't want to depend on external services to build and deploy their internal Python projects can choose to install both Car
and Engine
from their private package index as opposed to PyPI (another concrete dependency).
You can't merge these two types of dependencies together into one file without hampering people's ability to choose where to get their dependencies from.
Does that clarify why setup.py
/pyproject.toml
and requirements.txt
/Pipfile
need to be different things?
from pipfile.
@nchammas Thanks for the clarification! So basically, the workflow now is:
setup.py
/pyproject.toml
(abstract dependencies / version specs (like>= 1.8.0, < 2.0
)) — a lot of these; one for each library we depend on- Pipfile (where to get them; may be different between forks)
- Pipfile.lock (which exact versions to get).
Did I get it right?
What happens if setup.py
gets updated? Pipfile gets out of date, we need to sync it; how do we do it?
Example: imagine the following situation:
# Car's Pipfile
with source("https://super-secret-cheeseshop.pythonlabs.com/", verify_ssl=True):
package("Engine", ">=1.0.0")
# Engine's setup.py
setup(
name="Engine",
version="1.0.0",
packages=["engine"],
)
At some point Engine's author decides to update it:
# Engine's new setup.py
setup(
name="Engine",
version="1.1.0",
packages=["engine"],
install_requires=["Cylinder >= 1.1"],
)
Now we do pip freeze -p Pipfile
. Where do we get Cylinder from: the PyPI or the super secret cheeseshop?
from pipfile.
@askabelin If a project isn't meant to be importable, you don't need a setup.py at all, and can just use requirements.txt or a Pipfile. Yes, a setup.py can be used for the application-or-service use case, but the extra complexity isn't typically needed in such cases, and may give folks the misleading impression that the component is intended to be usable as a library.
from pipfile.
@iamale - The difference is explained here: https://caremad.io/posts/2013/07/setup-vs-requirement/
from pipfile.
@nchammas pip (requirements.txts) does not currently support specifying where a specific package comes from.
It just lets you specify globally your set of indexes.
from pipfile.
May be I am missing something but this discussion doesn't answer the question "what can go wrong if you'll put your app's concrete dependencies in setup.py and you don't have abstact ones". Not a good practice. Abstract matters. It is still not enough for data duplication in my opinion.
This split between abstract and concrete is an important one. It was what allows the PyPI mirroring infrastructure to work. It is what allows a company to host their own private package index.
Why do we need "abstract" deps if app is not supposed to be imported in other projects and to be uploaded to public PyPI (because it is private app)? It is not a problem to deploy it from private PyPI as well.
A project can have only have 1 set of abstract dependencies (setup.py/pyproject.toml), but different users working with that project can have different sets of concrete dependencies.
Different sets of dependencies (dev/ops) can be resolved with extras_require
. People can play with versions but versions that work for sure can be specified in setup.py. What is the purpose of abstract dependencies if application was not tested with it? Just "should work".
I agree that setup.py is a bad place for index url. Index is more related to infrastructure than to code. So it could be provided externally with Ansible / SALT etc.
May be someone has real use case where just setup.py was a problem. Please share it.
from pipfile.
libraries just don't need to commit their lockfile.
Indeed, they don't have to, since for building/installation setup.py
is sufficient. However, as developer of the library you might want to share your environment for testing purposes. Pipfile.lock
would provide a file format for that use (like requirements.txt
now). And for when deploying an environment.
from pipfile.
To provide a really practical example, I have a project at work where we do a few things.
- setup.py gets necessary app dependencies
- requirements.txt is used for defining an internal pypi and adding runtime deps (db driver to use)
- dev_requirements.txt is used to install test/dev reqs like pytest, mock, webtest, etc.
- ops_requirements.txt is used for a parallel operational venv that became necessary b/c a lib conflicted with the app requirements!
This project is for some integration so its requirements might be complicated than others, but generally, keeping app requirements you need in setup.py and runtime or deployment requirements in requirements.txt ends up helpful for the simple reason you can use pip flags in requirements.txt.
from pipfile.
That's a good question. I'm not sure what the answer is, but we could probably find out by testing this scenario with requirements.txt
, since Pipfile
is filling the same role.
@dstufft / @ncoghlan: If I instruct pip to install a package from a specific source (e.g. a private index), does it automatically try to install all that package's dependencies from the same source?
Or would it grab just the specifically mentioned package from the private index, and everything else. including dependencies of that package, from the default source (PyPI)?
from pipfile.
Unless OP (or someone else) has any more questions regarding why there's a proposal to have 3 separate files in the Python Packaging System to manage dependencies, let's close this issue.
from pipfile.
I'm going to go ahead and close this. There doesn't seem to be anything actionable here. If I'm wrong please speak up or open another issue, thanks.
from pipfile.
@k0nserv : You mentioned there being a .crate
file in Cargo that's roughly equivalent to setup.py
/ pyproject.toml
, .gemspec
, .podspec
, etc, but that doesn't line up with what I know or what I can find. All I can find about .crate
is the files that are uploaded to crates.io (like .whl
files in Python, IIUC).
Can you help clear up my confusion?
If it's true that there's no equivalent to those files in Cargo, that seems like a signal that we may wish to analyze. Cargo's constraints are different enough from ours that there might be some reason that we need the distinction, but if the designers of Cargo, having already lived with the design of .gemspec
, Gemfile
, and Gemfile.lock
, decided to not emulate it, then that may be a split that we ultimately may not want to make either.
This article on the gem files, as well as this comment from @dstufft about the utility of the split, makes sense to me. I just wonder if there's something that the folks that made Cargo know that we don't, and what they'd recommend to us.
http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/
jazzband/pip-tools#418 (comment)
from pipfile.
@ryanhiebert Yes you are correct. Rust and Cargo uses Cargo.toml
to specify both libraries and binaries. However as in other languages when building a library to be consumed by others you don't include Cargo.lock
in the project instead relying on transitive dependencies being installed when the library itself is installed.
from pipfile.
One of the folks that made Cargo is the person who wrote that article :) One of the main differences though is that there isn't really a runtime declaration with Cargo. Everything is build time and Cargo produces a static binary. That changes the math a bit.
from pipfile.
Yeah, I knew that he was involved with Cargo, which seems like it may be evidence that we need to consider that choice, since Cargo was made well after that post was written. Since he knew the benefits and pitfalls of using it in Ruby at the time, his choice to not replicate it for Rust means something to me.
It makes sense to me that the compilation nature of Rust could change the math, but I haven't come up with how it changes it. It seems like virtually all of the reasoning that we have behind doing the three-file approach still applies, even to Rust.
My own reasoning at the moment really wonders if having Cargo.toml
fill both the slot of pypackage.toml
and Pipfile
is actually a bad choice. In my own limited work with Rust, I've created a library that included a reference implementation binary that used that library. The library shouldn't have the lock file checked in, but the binary reasonably should. So I had to choose, and had to go with keeping the lock file out of the repository so I didn't screw up the usage of it as a library. I could make them separate repositories, but that didn't seem worthwhile.
Thus, my own thinking makes me think that the split is a good thing. I wonder if we could get @wycats to comment on the factors that lead to the decision that Cargo made.
Perhaps for Cargo several of the things that are constraints for us were deemed unnecessary. If enough of our reasons for keeping them separate were deemed of enough less utility for Cargo, then the overhead of an extra file may have been enough to decide to take the bad with the good, and skip the added file.
from pipfile.
Since it's much easier to add features than it is to remove them, I'd advise any new packaging system to start with a single file the way Rust has, and then use experience-driven-pain to decide whether or not they need anything more complex than that.
For the specific case you gave, it seems that Rust has first class "example application" support built (http://doc.crates.io/manifest.html#examples) and otherwise requires that applications and libraries by laid out differently (http://doc.crates.io/manifest.html#the-project-layout). It also has workspaces to more explicitly manage scenarios where projects are developed and published as multiple crates in a single repository: http://doc.crates.io/manifest.html#the-workspace-section
The big difference in Python is that we have two rather distinct problems to solve:
- how to provide an opinionated and coherent publisher-centric dependency management experience for folks starting a new project or actively changing the workflow for an existing one (i.e.
pipenv
) - how to provide a consumer-centric "minimum required build bootstrapping capability" that can be readily incorporated into existing build and deployment tools (i.e.
pyproject.toml
)
As a new ecosystem, Rust had the luxury of imposing Cargo.toml
and Cargo.lock
as their publishing standard from the start, rather than Python's more complicated history of retrofitting first distutils (~1998), then setuptools (~2004), then virtualenv (~2007), then pip (~2011), and now pyproject.toml and pipenv (~2017).
from pipfile.
Since it's much easier to add features than it is to remove them
I think that's the motivation I was looking for. Makes perfect sense to try a more limited approach first, then expand if necessary. Thank you. Clearly, Python is way past that already.
from pipfile.
I followed up with a concrete proposal in #98 that I believe gives us something actionable and pragmatic that could improve DX for maintaining Python libraries.
from pipfile.
Related Issues (20)
- Link additional Pipfiles from root one
- Skip lock on a single dependency HOT 4
- Pipfile.lock() is not idempotent
- Security Vulnerability Alerts (in examples/Pipfile.lock) HOT 5
- Pipenv take wrong Pipfile
- AttributeError while using inject_environment_variables function
- typo in line 136
- Rename 'master' branch to 'main' HOT 3
- pip2 install python
- - [ ] @monkeyhammercopiseded #165 HOT 5
- pipfile-spec in Pipfile
- Docs url is unreachable
- Publish a wheel for `pipfile`
- how to specific version in extras of pipfile?
- Support git+ssh source package repositroy
- ااا
- http://example.edu/tt
- Ggg
- how to add dependencies from other sources instead of pypi? HOT 1
- OSS-Fuzz Integration Inquiry
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pipfile.