migrated from https://bitbucket.org/pypa/python-packaging-user-guide/issue/27/quick-recommendations-page-obstacles-to
Graham Wideman created an issue 2013-10-23
On another recent issue I've commented that I'm trying to assemble coherent recommendations for packaging and deployment doc improvements, and that is still in process.
In the meantime, a couple of most-pressing issues with the Quick Recommendations page, which, if clarified either on that page or here, would help at least me considerably.
A) Quick Recommendations appears to recommend a non-functional process:
setuptools for packaging
setuptools docs says setuptools produces eggs
pip for installing such a package
Quick Recommendations page and also the pip docs says that pip cannot install eggs.
How should these conflicting assertions be reconciled?
B) Quick Recommendations recommends a process that seems a big leap from plain manual copying files around.
An appreciable portion of the audience for this page will be moving up from simply copying their script and module files from one machine to another manually, and then manually fixing up sys.path, PYTHONPATH, .pth files or whatever.
They are interested in doing this process in a less manual, more repeatable and reliable fashion, and don't necessarily have any use for (and might want to deliberately avoid) sending their distributables through some server (PyPI).
Is it safe to assume that the deployment tools can produce a distributable locally, and can consume a distributable locally? I think this scenario should be prominently mentioned, with discussion of whether or not the setuptools + pip satisfies this scenario.
Comments (11)
Graham Wideman
I've finally got to somewhat of a grip on a set of conceptual problems that are distributed across PPUG, and the PyPA Setuptools and pip docs, and the python.org docs on Distributing and Installing (aka Distutils) docs.
A) The advice on the Quick Recommendation page to "Use Setuptools to build and package" takes the reader off to investigate Setuptools. PyPA Setuptools doc advertises that that it creates eggs. But then pip docs says that pip cannot install eggs.
What's not evident from this sequence of reading is that:
Counter to naive expectations, the fact setup.py calls setuptools.setup() does not imply that setuptools is providing the API that the user will encounter, and does not imply that the Setuptools docs are sufficient to understand how to "Use Setuptools".
I ended up resorting to reading the setuptools source to find out that setup.py's call to setuptools.setup() is actually a call to distutils setup(), hence the command vocabulary is that of distutils, except for the commands added by setuptools.
So "Use Setuptools" really means "use Distutils maybe with a few features that are added in by Setuptools"... and therefore read the Distutils docs to figure out what is going on.
In particular, the advice to "use Setuptools" does not imply "invoke Setuptools to produce the one archive format that Setuptools docs tell how to produce."
So rather crucial on the current page is that "Use Setuptools" should be replaced by "Use Setuptools, and at the 'create archive' step use the sdist command supplied by underlying distutils so that the resulting archive is in "source distribution" form.' or whatever the correct course of action is.
(FWIW, I have noted that sdist appears in the example on the Packaging Guide page. Confusingly, although the example setup.py shows import from setuptools, the discussion under "Create your first release" proceeds as though setup is calling distutils directly, and setuptools is discussed later as an alternative thing to do to get more functionality.)
B) The comment "Use easy_install or buildout if you need to install from the binary egg format, which pip can’t currently handle" is ambiguous. It may well suggest (as I originally read it days ago) that what pip can't handle is the binary variant of egg format, but otherwise pip can handle "non binary" egg format. Of course, that leads to speculation of what the "binary version of egg format" might be -- perhaps an egg in which the files are compiled rather than pure python source code.
Based on my current understanding (which still could be wrong) that pip can't install any eggs, this statement should be revised to remove the word 'binary' since it adds nothing and just makes the sentence ambiguous.
C) Although not solely the fault of this page per se, a great deal of ambiguity results from virtually every page related to this topic failing to be clear about specifically what files and formats a particular tool produces or consumes. As a minimal remediation, this Quick Recommendations page should recommend what archive format(s) to use (and why).
Examples of problem PyPA docs in this regard:
[http://docs.python.org/3.3/distutils/commandref.html#creating-a-source-distribution-the-sdist-command] vaguely says it produces a "source distribution", whatever that means. It probably has a specific meaning in the mind of the writer, but not for the reader.
http://pythonhosted.org/setuptools/setuptools.html, with the heading "Building and Distributing Packages with Setuptools", says that Setuptools produces eggs. No mention that when seemingly invoking Setuptools you are free to use the commands of Distutils and might well produce other formats of distributable.
http://www.pip-installer.org/en/latest/index.html the pip doc says vaguely ""A tool for installing and managing Python packages", and I found no other page documenting what format packages it digest. The only docs are conflicting statements that pip does NOT install eggs (http://www.pip-installer.org/en/latest/other-tools.html), and examples and flag docs showing that pip DOES install eggs. (http://www.pip-installer.org/en/latest/usage.html, http://www.pip-installer.org/en/latest/logic.html). That inconsistency of doc seems unusable for what's intended to be the recommended way of doing things.
So, beyond the scope of the current page, what the reader needs somewhere is a reliable discussion of what python distribution formats are in common use, and which tools work with them. And each tool's doc needs to provide a clear specification of what formats it produces or consumes.
Mark as spam Delete 2013-10-24
Marcus Smith
thanks for writing all this out. this is very helpful to see how people interpret things. I'll respond to all your points. may be this weekend though.
Edit Mark as spam Delete 2013-10-24
Marcus Smith
I can already say that we need to be keeping a list of todo's for the pip and setuptools docs (with setuptools having the biggest list). It's clearly not just about writing this guide.
Edit Mark as spam Delete 2013-10-24
Marcus Smith
First off, note that I just updated the Quick Rec page, but here's specific responses.
setuptools docs says setuptools produces eggs pip for installing such a package Quick Recommendations page and also the pip docs says that pip cannot install eggs. How should these conflicting assertions be reconciled?
yes, the setuptools docs needs to emphasize how it's really being used by devs, as an sdist generator (not an egg generator). I've created a dedicated issue to track changes we need to make to setuptools docs. https://bitbucket.org/pypa/python-packaging-user-guide/issue/30/modernize-and-improve-setuptools-docs
Is it safe to assume that the deployment tools can produce a distributable locally, and can consume a distributable locally? I think this scenario should be prominently mentioned, with discussion of whether or not the setuptools + pip satisfies this scenario.
yes, certainly. not sure right off where this should be mentioned, but I'll cut a ticket for it.
So "Use Setuptools" really means "use Distutils maybe with a few features that are added in by Setuptools"... and therefore read the Distutils docs to figure out what is going on.
I'm thinking setuptools should go ahead and fill out sections for all the commands for completeness, even the ones it doesn't alter, and just link, when it's not altering anything. That will make it easier on users to understand the layering.
suggest (as I originally read it days ago) that what pip can't handle is the binary variant of egg format, but otherwise pip can handle "non binary" egg format.
Egg is a binary distribution format. Not sure where you're getting the "non binary" egg format from?
Maybe you're talking about "egg-info" directories? Internally, pip actually uses setuptools to install from sdist. By default, setuptools would convert sdists to eggs and install them as eggs, but pip passes certain options that makes setuptools do the installation with the packages exposed and using "egg-info" dirs, not fully encapsulated eggs. That could be explained better here http://www.pip-installer.org/en/latest/logic.html#setuptools-pkg-resources I'll add an issue to pip for that.
examples and flag docs showing that pip DOES install eggs. (http://www.pip-installer.org/en/latest/usage.html,
you mean the VCS examples with "egg="? that's not installing from eggs, but point taken, we need to explain the origin of that syntax. I'll add a pip issue for that too.
Edit Mark as spam Delete 2013-10-27
Nick Coghlan
pull request #10 is somewhat related to this - it splits the quick recommendations into "these aren't going away" stable recommendations and "we aren't happy with the state of these, but they're the best current option" interim recommendations.
Mark as spam Delete 2013-11-02
Nick Coghlan
Reading Graham's comments more closely, I realised the tool split is actually less closely related than I thought ( commit 2b1ad9ff758d568087752853f5bea0447eb58ce2 splits the installation and packaging tool recommendations).
Agreed there's still a lot of work to be done on the build tooling side.
Mark as spam Delete 2013-11-02
Graham Wideman
I see that the new wording (as of 2013-11-03) is clearer. (And FWIW bypasses the ambiguity I'd stumbled over about "binary Egg format.)
Still mysterious I think to people just trying to get started on reliably distributing their Python code to colleagues:
Do distutils, setuptools, pip etc do anything for you if all you want to do is get your code from dev machine A to work on user machine B, and you don't want to futz with PyPI as intermediary?
Does "build" mean anything for distributing python projects as source code? (And if so, what?)
And a comment on footnote [3] regarding pyvenv instead of virtualenv: What's behind the advice to wait until 3.4 to use pyvenv, when pyvenv is in 3.3? I've used pyvenv in 3.3 and it sets up a virtual environment... but maybe I'm missing something?
Mark as spam Delete 2013-11-04
Nick Coghlan
"pip wheel" is useful for internal redistribution (since other people won't need build dependencies). For pure Python apps, it's probably easier to just chuck everything needed into a zip archive with a main.py file (that's why that feature was added back in Python 2.6 - with any luck we will have better support for creating such archives in Python 3.4)
The build step can be useful even for pure Python projects if they distribute command line scripts. It also deals with various metadata related things
The main reason to avoid pyvenv in 3.3 is the fact it doesn't bootstrap pip automatically, whereas virtualenv does.
Mark as spam Delete 2013-11-04
Graham Wideman
Hi Nick -- followup:
Again regarding build: I am very familiar with what "build" normally means for other environments, like Java. But in Python packaging usage, it's not clear (at least to me) what the word "build" is intended to encompass. Does it refer to steps taken before deployment (as in preparing a binary distribution), or after distribution (like compiling, on the user's machine, from a source distribution), or both? And is it generally reckoned that there's a build step on both developer's and user's machines? For both source and binary distributions?
Hmmm "Build" would be a great term to have in the Glossary!
Regarding chucking everything into a zip: Yes, but colleagues may still need 3rd party dependencies to be fetched. Also: "that's why that feature was added back in Python 2.6". What feature are you referring to? Some setuptools or distutils command? Or something else?
Thanks.
Mark as spam Delete 2013-11-04
Nick Coghlan
Agreed, build would be a good glossary term (it refers to doing anything starting from source, whether creating a wheel file or installing directly on the target system). When wheels are used, there's no build step on the deployment target, so pip can also skip installing the build dependencies.
The feature I'm referring to is CPython's ability to execute a zip archive with a top level main.py file as a script, implicitly adding the zip file as the first entry on sys.path so you can import from it. For scripts with only pure Python dependencies, that's a quick and easy way to make them redistributable without needing an install step.
Mark as spam Delete 2013-11-05
Graham Wideman
Hi Nick: Thanks a lot for your comments.
So "build" specifically doesn't refer to steps that are just copying files to the right locations, or adding/changing .pth files, launch scripts/exes and so on?
On the python-executable zip (using main.py): OK, I hadn't previously noted that scheme. I guess that main.py could be used directly as the entry point for an application (the other modules of which are in the zip), OR main.py could be the entry point for the installer of whatever else is in the zip?
Is there a tool that creates either of these kinds of distributable? Or is the idea to create these manually (or with DIY scripts)?