jonasalfredsson / docker-devpi

Docker images of the Devpi server and client program, which work as a local index and caching server for pip packages.

docker-devpi

Docker images for both devpi-server and devpi-client, with both Debian and Alpine versions available.

Devpi allows you to host a local PyPi cache, along with the ability for you to add your own packages that you do not want to upload publicly.

ℹ️ Docker tags use the same version numberings as the devpi packges on PyPi (server & client), but can be viewed on DockerHub as well: server & client.

Usage

This repository is split into two parts: the server and the client. You will of course need a working server before there is any point in using the client.

Server

Environment Variables

• DEVPI_PASSWORD: The password to set for the "root" user on init (required).

Run it

The DEVPI_PASSWORD environment variable will set the root password on the first startup of this image. Set it to something more secure than what I use as an example here.

docker run -it -p 3141:3141 \
    -e DEVPI_PASSWORD=password \
    -v $(pwd)/data-server:/devpi/server \
    --name devpi-server jonasal/devpi-server:latest

The command will also host mount the data directory from the server to your local folder, in order to persist data between restarts.

The first time this container is started a full re-index of the upstream PyPi will commence, so the logs might be spammed with "Committing 2500 new documents to search index." for a while.

It is possible to customize the devpi instance with any of the available arguments by appending them to the command displayed above:

docker run -it <...> jonasal/devpi-server:latest \
    --threads 4 \
    --debug

Client

Environment Variables

• DEVPI_URL: The URL to the devpi instance to connect to (default: http://localhost:3141/root/pypi)
• DEVPI_USER: The username to login to the devpi instance with (default: "")
• DEVPI_PASSWORD: The password for the user when logging in (default: "")

Run it

The DEVPI_URL environment variable is the only one that needs to be set, and it will use the default unless you specify something else. The other two are optional, and can be defined for convenience so you do not have to manually login every time you start this container. Furthermore, if DEVPI_USER is set but DEVPI_PASSWORD is empty you will be prompted for the password.

The following command expects that there is a functional devpi instance running on the local computer, and that the "root" user has a password set. The host mounted directory may be used to transfer files in and out of the container.

docker run -it --rm --network host \
    -e DEVPI_USER=root \
    -e DEVPI_PASSWORD=password \
    -v $(pwd)/data-client:/devpi/client \
    jonasal/devpi-client:latest

Important note here is that this container uses the "host" network, else the localhost in the URL would just make so that the container tried to contact itself and would thus not reach the container running the server program.

Configure pip

For a quick test to see if the devpi-server actually works you can make a one-off installation like this:

pip install -i http://localhost:3141/root/pypi simplejson

The server logs should move and the pip installation should be successful.

To then make this setting a bit more permanent you can edit ~/.pip/pip.conf with this:

[global]
index-url = http://localhost:3141/root/pypi

Following installations should then be going through your local cache.

Set Up Local Index

By default devpi creates an index called root/pypi, which serves as a proxy and cache for PyPI, and you can’t upload your own packages to it.

In order to upload local packages we need to create another index, and to make our lives easier we will also configure it so that it "inherits" from the root/pypi index. What this meas is that if our search for a package in the local index fails it will fall back to the root/pypi index and try to find it there. Thus we can add all our private packages without losing the ability to find all the public ones.

To achieve this we will start the devpi-client container as root (we need to be able to make modifications):

docker run -it --rm --network host \
    -e DEVPI_USER=root \
    -e DEVPI_PASSWORD=password \
    -v $(pwd)/data-client:/devpi/client \
    jonasal/devpi-client:latest

Inside the container we will first create another user:

devpi user -c local password=something_long_and_complicated

Then we will create an index under this new user that inherits from the root/pypi index:

devpi index -c local/stable bases=root/pypi volatile=False

Restart the container again, but this time specify the new index URL and the new user:

docker run -it --rm --network host \
    -e DEVPI_URL="http://localhost:3141/local/stable" \
    -e DEVPI_USER=local \
    -e DEVPI_PASSWORD=something_long_and_complicated \
    -v $(pwd)/data-client:/devpi/client \
    jonasal/devpi-client:latest

You can now upload files from the /devpi/client folder to the current index by running something similar to this:

devpi upload /devpi/client private_package-0.1.0-py3-none-any.whl

After this you should be able to install it by pointing to the new index:

pip install -i http://localhost:3141/local/stable private_package

Upgrading

⚠️ This is semi-experimental since I am having trouble finding official documentation on the proper way to do this.

The best guide on upgrading devpi server I could find was this one, so I tried to integrate that into the entrypoint. It is not very automated, but you will at least have full control of what it is doing the entire time.

ℹ️ You should also only have to do this procedure when there is a change in the database schema. Such a change will warrant a major version bump, so this process is not necessary if you are just upgrading a minor or a patch version.

Begin by stopping any of the currently running devpi containers, and then run the same image again with the /export folder mounted to initiate an export process:

docker run -it --rm -e DEVPI_PASSWORD=password \
    -v $(pwd)/data-server:/devpi/server \
    -v $(pwd)/tmp:/export \
    jonasal/devpi-server:old-tag

If this is successful you should now rename the old data folder (don't delete it before you know the new one works):

sudo mv data-server data-server.bak

Run the new image with the /import folder mounted in order to initiate an import process:

docker run -it --rm -e DEVPI_PASSWORD=password \
    -v $(pwd)/data-server:/devpi/server \
    -v $(pwd)/tmp:/import \
    jonasal/devpi-server:new-tag

When this one completes you can go back to running the image normally without any of the import/export folders mounted. This should most likely give you a functional upgraded instance of devpi. Cleanup of the extra folders can be done with this simple command

sudo rm -r data-server.bak && sudo rm -r tmp

Further Reading

I got most of the information I needed to complete this project from the following sources, perhaps they are useful for you too:

• Devpi Quickstart Guide
• Stefan Scherfke: Getting started with devpi
• @kyhau: devpiNotes.md
• Mpho Mphego: How I Setup A Private Local PyPI Server Using Docker And Ansible
• @kyhau: devpiServerUpgrade.md

docker-devpi's Issues

[Server] Source *.rc files in /entrypoint.sh

Entrypoint plugins are currently not able to change the startup environment. I think it'd make sense to add a *.rc case and then source the files found instead of executing them.