python-packaging-tutorial / python-packaging-tutorial

Tutorial on python packaging

python-packaging-tutorial

Tutorial on python packaging at SciPy 2018

Rendered notes here:

https://python-packaging-tutorial.readthedocs.io

Installation instructions

Windows

• Install Miniconda3: https://conda.io/docs/user-guide/install/windows.html
• Install conda-build using conda: https://conda.io/docs/user-guide/tasks/build-packages/install-conda-build.html
• install Visual Studio 2017: https://www.visualstudio.com/thank-you-downloading-visual-studio/?sku=Community&rel=15&utm_source=vscom&utm_medium=clickbutton&utm_campaign=tailored_featurepgcppsp&rid=34346
• Install Docker for Windows: https://www.docker.com/docker-windows
• create a conda environment for playing with scikit-build: conda create -n skbuild -c conda-forge cmake scikit-build cython

MacOS

• Install Miniconda3: https://conda.io/docs/user-guide/install/macos.html
• Install conda-build using conda: https://conda.io/docs/user-guide/tasks/build-packages/install-conda-build.html
• Install Xcode: ​https://itunes.apple.com/us/app/xcode/id497799835?mt=12​ .
• Follow instructions in the conda-build documentation to install an older MacOS SDK: https://conda.io/docs/user-guide/tasks/build-packages/compiler-tools.html#macos-sdk
• Install Docker for Mac: https://www.docker.com/docker-mac
• create a conda environment for playing with scikit-build: conda create -n skbuild -c conda-forge cmake scikit-build cython

Linux

• Install Miniconda3: https://conda.io/docs/user-guide/install/linux.html
• Install conda-build using conda: https://conda.io/docs/user-guide/tasks/build-packages/install-conda-build.html
• Install Docker client for your OS:
  • Ubuntu: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04
  • Fedora/RHEL: https://developer.fedoraproject.org/tools/docker/docker-installation.html
• create a conda environment for playing with scikit-build: conda create -n skbuild -c conda-forge cmake scikit-build cython

Docker Image caching

• run docker pull dockcross/manylinux-x64

Testing installation

• Clone this repository
• cd into the test_recipes folder
• run conda build bitarray. Success is indicated by output like:

===== bitarray-0.8.1-py36h1de35cc_1 OK =====
import: 'bitarray'

Resource usage statistics from testing bitarray:
   Process count: 1
   CPU time: Sys=0:00:00.0, User=0:00:00.0
   Memory: 1.1M
   Disk usage: 16B
   Time elapsed: 0:00:02.0

TEST END: /Users/msarahan/miniconda3/conda-bld/osx-64/bitarray-0.8.1-py36h1de35cc_1.tar.bz2
Renaming work directory,  /Users/msarahan/miniconda3/conda-bld/bitarray_1529267981928/work  to  /Users/msarahan/miniconda3/conda-bld/bitarray_1529267981928/work_moved_bitarray-0.8.1-py36h1de35cc_1_osx-64_main_build_loop
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/bitarray-0.8.1-py36h1de35cc_1.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []
####################################################################################
Resource usage summary:

Total time: 0:00:36.9
CPU usage: sys=0:00:00.2, user=0:00:00.2
Maximum memory usage observed: 50.6M
Total disk usage observed (not including envs): 1016B


####################################################################################

• cd into the test_recipes/hello-cython folder
• activate the skbuild environment: conda activate skbuild. For older conda (<4.4) installations, follow legacy instructions at https://conda.io/docs/user-guide/tasks/manage-environments.html#activating-an-environment
• run python setup.py install. Success is indicated by output ending in lines like:

creating 'dist/hello_cython-1.2.3-py3.5-macosx-10.9-x86_64.egg' and adding '_skbuild/setuptools/bdist.macosx-10.9-x86_64/egg' to it
removing '_skbuild/setuptools/bdist.macosx-10.9-x86_64/egg' (and everything under it)
Processing hello_cython-1.2.3-py3.5-macosx-10.9-x86_64.egg
Copying hello_cython-1.2.3-py3.5-macosx-10.9-x86_64.egg to /Users/msarahan/miniconda3/envs/skbuild/lib/python3.5/site-packages
Adding hello-cython 1.2.3 to easy-install.pth file

Installed /Users/msarahan/miniconda3/envs/skbuild/lib/python3.5/site-packages/hello_cython-1.2.3-py3.5-macosx-10.9-x86_64.egg
Processing dependencies for hello-cython==1.2.3
Finished processing dependencies for hello-cython==1.2.3

• You're all set!

Troubleshooting

• Please file issues on this github issue tracker: https://github.com/python-packaging-tutorial/python-packaging-tutorial/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc

Package hosting service accounts

Pprior to attending the tutorial, participants should set up an account on

• PyPI (​https://pypi.org/account/register/) and TestPyPI (https://test.pypi.org/account/register/)
• anaconda.org (​https://anaconda.org/​)

python-packaging-tutorial's Issues

Format / tools for presentation, etc?

(I hope it's OK to use this repo for discussion and development of the tutorial)

What do we want to use to create the slides / materials for the tutorial?

I've had good luck with Sphinx and Hieroglyph -- Hieroglyph makes slides, and you also get a nice HTML page for folks to read -- not as good for presenting, but better for reading on your own.

Here's an example from my Python class:

http://uwpce-pythoncert.github.io/SystemDevelopment/packaging.html#packaging

Which is also my packaging talk, which may or may not be helpful when putting this together.

That is the regular html version, the slides version I don't have published, but the source is here:

https://github.com/UWPCE-PythonCert/SystemDevelopment/tree/master/slides_sources

If you have hieroglyph installed, you can simply "make slides" and get your html slides.

Continuous integration section

• scikit-ci: https://github.com/scikit-build/scikit-ci
• scikit-ci-addons: https://github.com/scikit-build/scikit-ci-addons
• Multibuild - https://github.com/matthew-brett/multibuild

License?

I see an Apache License in here now.

I'm OK with pretty much an Open Source License, but if I had a choice, I'd prefer the Creative Commons Share Alike License:

https://creativecommons.org/licenses/by-sa/3.0/us/

Others' thoughts?

Relocate "distribution building" slides from "setyup_py" to "uploading_pypi.rst"

@ChrisBarker-NOAA I am thinking of relocating these slides to "uploading_pypi.rst":

• Python packaging tools

• Where do I go to figure this out?

• Running setup.py: This slides would be simplified referencing only the "develop" mode and adding a link to the "uploading_pypi.rst" section

Let me know what you think

Add CONTRIBUTING document

Test examples

All examples should be tested

Reference to command, code snippet could be integrated using literalinclude.

For example:

  .. literalinclude:: ../appveyor.yml
     :language: yaml
     :start-after: scikit-ci-yml.rst: start
     :end-before: scikit-ci-yml.rst: end
     :emphasize-lines: 2, 5, 8, 11

The start-after / start-before are optional, they allow to limit the focus on a section of a script. In this case, appveyor.yml

Binaries: Reference manylinux, auditwheel and delocate

• Manylinux docker image - https://github.com/pypa/manylinux
• Delocate - https://github.com/matthew-brett/delocate
• Auditwheel - https://github.com/pypa/auditwheel

cc: @thewtex

Help needed with conda-build and setting up meta.yaml, and setup.py for module/submodule structure.

Hello,

I am trying to build a conda package, using conda-build, from pure python modules (and submodules). As a starter I use the cookiecutter at https://github.com/conda/cookiecutter-conda-python.git.
If I understand correctly, one can only include python submodules if the source files are on github ?
Can someone please point me to a comprehensive example (even for a non professional programmer) on how to make a pure python package using conda-build, out of a simple module/submodule structure ?
To be more precise, I'm interested in how to set :

• meta.yaml file (and how to use load_setup_py_data() function ?)
• setup.py file.
  For further information : I manage creating a package from a local file where all my python code is in one file, but when I use the same approach on a git source (where I have only my python code in modules and submodules) , I get this error :

(%BUILD_PREFIX%) %SRC_DIR%>python setup.py install --single-version-externally-managed --record=record.txt
python: can't open file 'setup.py': [Errno 2] No such file or directory

Thanks

Transition to readthedocs

Move "Installation instructions" and "Testing installation" to website

• Create sphinx_presentation/prerequisites.rst
• Move the related content from README.md into prerequisites.rst file
  • Make sure to convert to use restructure text convention by looking at other *.rst file in the sphinx_presentation folder
  • Make sure to prevent the content added to prerequisites.rst from being included in the presentation slides.

Slides and website can be generated reading these instructions: sphinx_presentation/README.rst

Create Logo