Coder Social home page Coder Social logo

devpi / devpi Goto Github PK

View Code? Open in Web Editor NEW
829.0 13.0 126.0 22.09 MB

Python PyPi staging server and packaging, testing, release tool

Home Page: https://doc.devpi.net

Python 99.06% Shell 0.07% JavaScript 0.29% CSS 0.58%
pypi server python packaging testing

devpi's Introduction

devpi: PyPI server and packaging/testing/release tool

This repository contains three packages comprising the core devpi system on the server and client side:

  • devpi-server: for serving a pypi.org consistent caching index as well as user or team based indexes which can inherit packages from each other or from the pypi.org site.
  • devpi-web: plugin for devpi-server that provides a web and search interface
  • devpi-client: command line tool with sub commands for creating users, using indexes, uploading to and installing from indexes, as well as a "test" command for invoking tox.

For getting started, more docs see https://doc.devpi.net/

Holger Krekel, Florian Schulze, April 2017 (contact us at mail at pyfidelity.com for support contracts and paid help)

devpi's People

Contributors

a-feld avatar andrewleech avatar avdn avatar chadawagner avatar chrullrich avatar designerzim avatar eachimei avatar evemorgen avatar fschulze avatar gotcha avatar hpk42 avatar jaraco avatar jonathanrrogers avatar keimlink avatar lbrack avatar ldx avatar matthias-bach-by avatar mattjhussey avatar michael-k avatar msabramo avatar pi-anl avatar pr0ps avatar rodcloutier avatar santegoeds avatar stephanerb avatar tbartlett0 avatar userzimmermann avatar vytas7 avatar xolox avatar zy1o avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

devpi's Issues

devpi upload : error handling for package in a hg repository but not under revision control

If the package is within an hg repository but not tracked, devpi upload will fail with a backtrace has oppose to providing a meaningfull error message (granted this is a corner case)

#!python

$ devpi upload  --format sdist 
created workdir /tmp/devpi6
--> $ hg st -nmac .
hg-exported project to <Exported /tmp/devpi6/upload/archimede>
--> $ /home/lpbrac/bitbucket/devpi_doc_contrib/bin/python /home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/upload/setuppy.py /tmp/devpi6/upload/archimede http://localhost:3141/emilie/dev/ emilie emilie-97481e0b042b8cf3f6d4a1caec7abd5073ea454b62d11f2bf3286d24b283904f.BOWvJQ.rBIYcxatgCxLHvyDeQ6Z3a7MVSU register -r devpi
Traceback (most recent call last):
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/bin/devpi", line 9, in <module>
    load_entry_point('devpi-client==0.9.5.dev6', 'console_scripts', 'devpi')()
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/main.py", line 35, in main
    return method(hub, hub.args)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/upload/upload.py", line 41, in main
    exported.setup_register()
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/upload/upload.py", line 280, in setup_register
    cwd = self.rootpath)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/main.py", line 215, in popen_output
    return check_output(args, cwd=str(cwd))
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/main.py", line 49, in check_output
    popen = Popen(stdout=PIPE, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: '/tmp/devpi6/upload/archimede'

Most likely is that *hg st -nmac . does returns an empty list of file causing the staging area to be empty and therefore failing.

An appropriate error message should be provided.

devpi test - tox parametrization

We are currently integrating devpi with our build system and while being able to run devpi test works well in most cases, we would like to pass parameters to the underlying tox process via the devpi test command.

That is being able to do something like:

#!bash

devpi test --tox-param="<parameters>"

Thank you for considering this proposal.

devpi use with --venv fails with unhandled exception if the venv doesn't exit

When calling devpi use with --venv and this environment doesn't exist (was not created with devpi install, the following exception occurs and is not handled.

#!python

$ devpi use http://localhost:3141/laurent/dev --venv sandbox
Traceback (most recent call last):
  File "/usr/local/bin/devpi", line 9, in <module>
    load_entry_point('devpi-client==0.9.4', 'console_scripts', 'devpi')()
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 24, in main
    return method(hub, hub.args)
  File "/usr/local/lib/python2.7/dist-packages/devpi/use.py", line 132, in main
    cand = hub.path_venvbase.join(venvname, vbin)
AttributeError: 'NoneType' object has no attribute 'join'

This case should be handled with an error message could not be found.

Server error after upgrading to 1.0

I just upgraded from devpi 0.9.2 to 1.0, and when uploading I get a 500 response and the following stack trace in the logs:

#!python

Traceback (most recent call last):
  File "/usr/local/bin/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/usr/local/bin/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/usr/local/lib/python2.7/dist-packages/devpi_server/views.py", line 421, in submit
    trigger_jenkins(stage, name)
  File "/usr/local/lib/python2.7/dist-packages/devpi_server/views.py", line 724, in trigger_jenkins
    jenkins_url = stage.ixconfig["uploadtrigger_jenkins"]
KeyError: 'uploadtrigger_jenkins'

I assume this is because the old index config didn't have the 'uploadtrigger_jenkins' value set, i.e., this is an upgrade issue.

TEST: devpi client testing : Fixture cmd_devpi not checking HTTP code correctly

I am adding some test to the devpi client and working in the test_functional file,
leveraging the Mapp class.

Each method of this class has a "code" parameter which I expected to be the "expected response" from the server, leading to an exception if a different HTTP return code was received.

It turns out that this is not the case. The fixture does the following:

#!python

        try:
            ret = method(hub, hub.args)
        except SystemExit as sysex:
            ret = sysex.args[0] or 1
        if ret and kwargs.get("code", 400) < 400:
            raise SystemExit(ret)

but it doesn't actually validate the code which is part of the response.

for instance, I am doing:

#!python

    def test_delete_root_forbidden(self, mapp):
        mapp.login_root()
        mapp.delete_user(user = "root", code=403)

which is in my opinion the expected behavior (according to issue #25) but the test pass even though I see on the standard output

#!bash

*** inline$ devpi --clientdir /tmp/pytest-46/test_delete_root_forbidden0/client user --delete root
200: user 'root' deleted

500 Error while installing chromelogger==0.3.0

Installation fails, pip.log is attached - not sure this is the same error as in #8

#!bash

devpi install chromelogger==0.3.0
--> $ tenv/bin/pip install -U -i http://localhost:3141/root/dev/+simple/ chromelogger==0.3.0
Downloading/unpacking chromelogger==0.3.0
  HTTP error 500 while getting http://localhost:3141/root/pypi/f/https/github.com/ccampbell/chromelogger-python/archive/0.3.0.zip#egg=chromelogger-0.3.0 (from http://localhost:3)
  Could not install requirement chromelogger==0.3.0 because of error HTTP Error 500: Internal Server Error
Could not install requirement chromelogger==0.3.0 because of HTTP error HTTP Error 500: Internal Server Error for URL http://localhost:3141/root/pypi/f/https/github.com/ccampbel)
Storing complete log in /home/kostia/.pip/pip.log
command failed

devpi use -l : exception when no user index created

Version: 1.0

The devpi use -l fails with an exception if a user exists on the server but has no associated index.

Steps to reproduce.

  • connect to existing instance
#!bash
$ devpi use http://localhost:3141
using server: http://localhost:3141/ (not logged in)
no current index: type 'devpi use -l' to discover indices
  • Check that the command works
#!bash
$ devpi use -l
root/pypi       bases=                volatile=False
  • create a user but no index
#!bash
$ devpi user -c user_bug password=1234
user created: user_bug
  • run the devpi use -l - this time fails when it hits user_bug
#!bash
$ devpi use -l
root/pypi       bases=                volatile=False
Traceback (most recent call last):
  File "/usr/local/bin/devpi", line 9, in <module>
    load_entry_point('devpi-client==1.0', 'console_scripts', 'devpi')()
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 29, in main
    return method(hub, hub.args)
  File "/usr/local/lib/python2.7/dist-packages/devpi/use.py", line 212, in main
    out_index_list(hub, r["result"])
  File "/usr/local/lib/python2.7/dist-packages/devpi/use.py", line 164, in out_index_list
    for index, ixconfig in indexes.items():
AttributeError: 'list' object has no attribute 'items'

install --venv fails with existing but empty directories

--venv <non_existing_dir> works, but --venv <empty_dir> does not. virtualenv itself however works in both scenarios. Imho, devpi which basically just wraps virtualenv in this case should behave the same.

I find this especially annoying in the devpi install --venv . case, which at the moment always fails, because "." obviously exists (otherwise I could not be in it ;) but may not yet be a virtualenv yet.

#!bash

$ devpi install --venv . pytest
no pip binary found

Upstream access for devpi-server does not work via proxy

The requests library already provides support for proxy server. Unfortunately the fiddling with the "Host" header entry when accessing front.python.org does not play with at least or proxy server. As I understand the last paragraph of section 14.23 of RFC 2616 (Hypertext Transfer Protocol -- HTTP/1.1):

An HTTP/1.1 proxy MUST ensure that any request message it forwards does contain an appropriate Host header field that identifies the service being requested by the proxy.

the proxy is required to modify the Host entry?

In a first step USE_FRONT in main.py should be set to "0" when a proxy is to be used (requests looks for an environment variable HTTPS_PROXY). If one day a "--proxy" option is added to devpi-server, using this option should also case USE-FRONT to be disabled.

devpi-client cannot set bases to empty set.

example:

#!python

(devpi)6:17:48 [email protected] ~ master ?  % devpi index
http://localhost:3141/test/dev/:
  type=stage
  bases=test/dev
  volatile=True
  uploadtrigger_jenkins=None
  acl_upload=test
(devpi)6:17:55 [email protected] ~ master ?  % devpi index dev bases=
dev changing bases:
PATCH http://localhost:3141/test/dev
400 Bad Request: invalid base index spec: u''
[1]    25667 exit 1     devpi index dev bases=
(devpi)6:18:01 [email protected] ~ master ?  % devpi index dev bases=test/dev                                           1 โ†ต
dev changing bases: test/dev
dev:
  type=stage
  bases=test/dev
  volatile=True
  uploadtrigger_jenkins=None
  acl_upload=test

devpi user -m : can not modify e-mail address HTTP Error 400

Password can be modified using this method but it is not possible to modify the email address.

#!bash
$ devpi login sophie --password 1234
logged in 'sophie', credentials valid for 10.00 hours
$ devpi user -m sophie [email protected]
PATCH http://localhost:3141/sophie
400 Bad Request: could not decode request

upload with expired login causes 500

"devpi upload" with an expired login causes this on the server side:

#!python

(22443) accepted ('127.0.0.1', 49450)
Traceback (most recent call last):
  File "/home/devpi/venv_deploy/bin/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/home/devpi/venv_deploy/bin/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/home/devpi/venv_deploy/lib/python2.6/site-packages/devpi_server/views.py", line 400, in submit
    self.auth.require_user(user, stage=stage)
  File "/home/devpi/venv_deploy/lib/python2.6/site-packages/devpi_server/views.py", line 114, in require_user
    auth_user = self.get_auth_user(request.auth)
  File "/home/devpi/venv_deploy/lib/python2.6/site-packages/devpi_server/views.py", line 89, in get_auth_user
    raise self.Expired()
Expired

devpi install - third level of inheritance doesn't seem to work

version: 0.9.4

I was not able to isolate the root cause via various experiments, but something doesn't seem right here. It appears that when setting an index to a user index other than root, installation of pypi package doesn't work

:Pre-requisite:

We have the following index tree:

#!bash

   /root/pypi 
       ^
       |
   /root/dev
       ^
       |
   /laurent/dev 

The json response looks like this

#!bash

$ devpi getjson http://localhost:3141
{
    "result": {
        "laurent": {
            "email": "[email protected]", 
            "indexes": {
                "dev": {
                    "bases": [
                        "/root/dev"
                    ], 
                    "type": "stage", 
                    "volatile": true
                }, 
            }, 
            "username": "laurent"
        }, 
        "root": {
            "indexes": {
                "dev": {
                    "bases": [
                        "root/pypi"
                    ], 
                    "type": "stage", 
                    "volatile": true
                }, 
                "pypi": {
                    "bases": [], 
                    "type": "mirror", 
                    "volatile": false
                }
            }, 
            "username": "root"
        }, 
    }, 
    "status": 200, 
    "type": "list:userconfig"
}

If I set the index to /laurent/dev and try to install pytest, the devpi install command will fail

#!bash

$ devpi use http://localhost:3141/laurent/dev --venv sandbox
using index:  http://localhost:3141/laurent/dev/
base indexes: http://localhost:3141/root/dev/
install venv: /home/lpbrac/devpi/sandbox
logged in as: laurent
$ devpi install pytest
--> $ sandbox/bin/pip install -U -i http://localhost:3141/laurent/dev/+simple/ pytest
Downloading/unpacking pytest
  Cannot fetch index base URL http://localhost:3141/laurent/dev/+simple/
  Could not find any downloads that satisfy the requirement pytest
No distributions at all found for pytest
Storing complete log in /home/lpbrac/.pip/pip.log
command failed

However, using /root/dev will work (obviously)

#!bash

$ devpi use http://localhost:3141/root/dev --venv sandbox
using index:  http://localhost:3141/root/dev/
base indexes: http://localhost:3141/root/pypi/
install venv: /home/lpbrac/devpi/sandbox
logged in as: laurent
$ devpi install pytest
--> $ sandbox/bin/pip install -U -i http://localhost:3141/root/dev/+simple/ pytest
Downloading/unpacking pytest
  Downloading pytest-2.3.5.tar.gz (426kB): 426kB downloaded
  Running setup.py egg_info for package pytest
...
    Installing py.test script to /home/lpbrac/devpi/sandbox/bin
    Installing py.test-2.7 script to /home/lpbrac/devpi/sandbox/bin
  Running setup.py install for py
    
Successfully installed pytest py
Cleaning up...

Even if I set the base of /laurent/dev directly to /root/pypi as its base (bypassing the second level), it still doesn't works:

#!bash

$ devpi index -m /laurent/dev bases=/root/pypi
201: Created
lpbrac@lpbrac-ubuntu-vm:~/devpi$ devpi getjson /laurent/dev
{
    "result": {
        "bases": [
            "/root/pypi"
        ], 
        "type": "stage", 
        "volatile": true
    }, 
    "status": 200, 
    "type": "indexconfig"
}
$ devpi use http://localhost:3141/laurent/dev --venv sandbox
using index:  http://localhost:3141/laurent/dev/
base indexes: http://localhost:3141/root/pypi/
install venv: /home/lpbrac/devpi/sandbox
logged in as: laurent

$ devpi install pytest
--> $ sandbox/bin/pip install -U -i http://localhost:3141/laurent/dev/+simple/ pytest
Downloading/unpacking pytest
  Cannot fetch index base URL http://localhost:3141/laurent/dev/+simple/
  Could not find any downloads that satisfy the requirement pytest
No distributions at all found for pytest
Storing complete log in /home/lpbrac/.pip/pip.log
command failed

QUESTION:

I am not quite clear about the meaning of --venv with the use sub command.
If I specify it, i see that the venv is set to sandbox but if I do a devpi use,
devpi claims that no current venv is set even though the virtual environment is activated.

#!bash

$ devpi use http://localhost:3141/laurent/dev --venv sandbox
using index:  http://localhost:3141/laurent/dev/
base indexes: http://localhost:3141/root/pypi/
install venv: /home/lpbrac/devpi/sandbox
logged in as: laurent
(sandbox)lpbrac@lpbrac-ubuntu-vm:~/devpi$ devpi use
using index:  http://localhost:3141/laurent/dev/
base indexes: http://localhost:3141/root/pypi/
no current install venv set
logged in as: laurent

devpi list: show upstream packages

version: 0.9.4

note: This issue is somewhat related to issue #17

The list sub-command only shows the package in the current index. It would be useful to list packages inherited from the bases. Currently, this process requires the user to
use all the bases and issue the list sub-command.

It would be useful to have a -p, --parents command line option which would also show the package inherited from the parent (adorned with the index path).

Since /root/pypi may contain a substantial amount of packages, the* -a, -all* used in conjunction with -p would list the packages cached from pypi.

#!bash

$devpi list 
my_package  

$devpi list --parent
my_package
my_dup          
base_1_package       /root/dev
base_2_package       /root/prod
my_dup               /root/prod

$devpi list --parent --all
my_package
my_dup          
base_1_package       /root/dev
base_2_package       /root/prod
my_dup               /root/prod
pytest               /root/pypi
nose                 /root/pypi
...

finally, the -s, --sort option would sort the list by package name (highlighting duplicates)

#!bash

$devpi list --parent --all --sort
base_1_package       /root/dev
base_2_package       /root/prod
my_dup          
my_dup               /root/prod
my_package
nose                 /root/pypi
pytest               /root/pypi

...

project name transformed upon upload

We came across a strange issue when uploading release files where the name contains _ in the name (in this case the name is gti_utils)

The setup file looks like this:

#!python

    setup(
            # Package name
            name              = 'gti_utils',
            version           = __version__,
            
            # Information about what needs to be packaged
            package_dir       = {'':'src'},
            packages          = find_packages('src'),
            scripts           = [],

The actual package name is gti_scutils. If I generate the package manually, I do get the right file::

#!bash

$ python setup.py sdist
running sdist
running egg_info
writing src/gti_utils.egg-info/PKG-INFO
writing top-level names to src/gti_utils.egg-info/top_level.txt
writing dependency_links to src/gti_utils.egg-info/dependency_links.txt
writing entry points to src/gti_utils.egg-info/entry_points.txt
reading manifest file 'src/gti_utils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/gti_utils.egg-info/SOURCES.txt'
...
Writing gti_utils-0.3.0rc/setup.cfg
Creating tar archive
removing 'gti_utils-0.3.0rc' (and everything under it)

But if I perform an upload with devpi I endup with the release file gti_utils-0.3.0rc registered under gti-utils. Devpi also creates an empty project with the appropriate name gti_utils.

When listing the index we find out this:

#!bash

$ devpi list
list result: http://devpi-eu.dolby.net/lpbrac/dev/
dolby-commander
dolby_commander
gti-utils
gti_utils
$ devpi list gti-utils
list result: http://devpi-eu.dolby.net/lpbrac/dev/
lpbrac/dev/gti_utils/0.3.0rc/gti_utils-0.3.0rc.tar.gz
(gti_utils)lpbrac@lpbrac-ubuntu-vm:~/p4ws/main/qa/tools/gti_utils$ devpi list gti_utils
list result: http://devpi-eu.dolby.net/lpbrac/dev/

The release file gti_utils-0.3.0rc.tar.gz (note the _) can be listed with gti-utils while gti_utils returns nothing.

Finally note the following. I can remove gti-utils but can't delete gti_utils as shown below:

#!bash
$ devpi remove gti-utils
About to remove the following release files and metadata:
   lpbrac/dev/gti_utils/0.3.0rc/gti_utils-0.3.0rc.tar.gz
Are you sure (yes/no)? yes 
$ devpi remove gti_utils
nothing to delete
$ devpi list 
list result: http://devpi-eu.dolby.net/lpbrac/dev/
dolby-commander
dolby_commander
gti_utils

devpi index index allows to use invalid index as a base

Version:

changeset: 257:acf0db041e3c
tag: tip
user: holger krekel [email protected]
date: Sun Aug 04 21:22:58 2013 +0200
summary: fix LICENSE to MIT (BSD was an accident)

while investigating issue #20, I realize that it is possible to specify a non valid index name leading to an exception on the server side.

here is the structure of my test index at the start:

#!bash

$ devpi getjson http://localhost:3141
{
    "result": {
        "emilie": {
            "email": "[email protected]", 
            "indexes": {
                "dev": {
                    "acl_upload": [], 
                    "bases": [
                        "/root/dev"
                    ], 
                    "type": "stage", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": true
                }
            }, 
            "username": "emilie"
        }, 
        "root": {
            "indexes": {
                "dev": {
                    "acl_upload": [], 
                    "bases": [
                        "root/pypi"
                    ], 
                    "type": "stage", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": true
                }, 
                "pypi": {
                    "acl_upload": [], 
                    "bases": [], 
                    "type": "mirror", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": false
                }
            }, 
            "username": "root"
        }
    }, 
    "status": 200, 
    "type": "list:userconfig"
}

I then modify the base with a invalid index name (eggplant) and the devpi server accepts the request.

#!bash

$ devpi index --modify /emilie/prod bases=/root/eggplant
200: OK
$ devpi getjson http://localhost:3141
{
    "result": {
        "emilie": {
            "email": "[email protected]", 
            "indexes": {
                "dev": {
                    "acl_upload": [], 
                    "bases": [
                        "/root/eggplant"
                    ], 
                    "type": "stage", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": true
                }
            }, 
            "username": "emilie"
        }, 
        "root": {
            "indexes": {
                "dev": {
                    "acl_upload": [], 
                    "bases": [
                        "root/pypi"
                    ], 
                    "type": "stage", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": true
                }, 
                "pypi": {
                    "acl_upload": [], 
                    "bases": [], 
                    "type": "mirror", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": false
                }
            }, 
            "username": "root"
        }
    }, 
    "status": 200, 
    "type": "list:userconfig"
}

When attempting to install a package (residing on PyPi), I get the following on the server side.

#!python

localhost - - [05/Aug/2013 16:21:56] "GET /emilie/dev/+simple/ HTTP/1.1" 500 59
AAAA emilie dev
AAAA /root/eggplant None
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/lib/python2.7/site-packages/bottle-0.11.6-py2.7.egg/EGG-INFO/scripts/bottle.py", line 874, in __call__
    return self.wsgi(environ, start_response)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/lib/python2.7/site-packages/bottle-0.11.6-py2.7.egg/EGG-INFO/scripts/bottle.py", line 849, in wsgi
    out = self._cast(self._handle(environ))
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/lib/python2.7/site-packages/bottle-0.11.6-py2.7.egg/EGG-INFO/scripts/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/lib/python2.7/site-packages/bottle-0.11.6-py2.7.egg/EGG-INFO/scripts/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/server/devpi_server/views.py", line 175, in simple_list_project
    result = stage.getreleaselinks(projectname)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/server/devpi_server/db.py", line 291, in getreleaselinks
    projectname=projectname):
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/server/devpi_server/db.py", line 194, in op_with_bases
    stage = self.db.getstage(base)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/server/devpi_server/db.py", line 140, in getstage
    user, index = user.split("/")
ValueError: too many values to unpack

The problem here is the initial '/' in /root/eggplant. If we work around this issue in the code, the invalid index results in an ixconfig set to None causing DB.op_with_base to fail with

note When fixing the issue of initial /, we actually resolve issue #20 allowing to install a pypi package via a user index.

#!python

  File "/home/lpbrac/bitbucket/devpi_doc_contrib/server/devpi_server/db.py", line 198, in op_with_bases
    base_entries = getattr(stage, opname)(**kw)
AttributeError: 'NoneType' object has no attribute 'getreleaselinks'

Note: when specifying the correct index

More elegant error message when malformed URL specified

When specifying an malformed URL, we currently get a back trace.

#!python

$ devpi getjson http://localhost:3141:/emilie
Traceback (most recent call last):
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/bin/devpi", line 9, in <module>
    load_entry_point('devpi-client==0.9.5.dev5', 'console_scripts', 'devpi')()
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/main.py", line 35, in main
    return method(hub, hub.args)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/getjson.py", line 24, in main
    r = hub.http_api("get", url, quiet=True)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/client/devpi/main.py", line 116, in http_api
    r = self.http.request(method, url, data=data, headers=headers)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/local/lib/python2.7/site-packages/requests-1.2.3-py2.7.egg/requests/sessions.py", line 324, in request
    prep = req.prepare()
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/local/lib/python2.7/site-packages/requests-1.2.3-py2.7.egg/requests/models.py", line 222, in prepare
    p.prepare_url(self.url, self.params)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/local/lib/python2.7/site-packages/requests-1.2.3-py2.7.egg/requests/models.py", line 288, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib/local/lib/python2.7/site-packages/requests-1.2.3-py2.7.egg/requests/packages/urllib3/util.py", line 156, in parse_url
    raise LocationParseError("Failed to parse: %s" % url)
requests.packages.urllib3.exceptions.LocationParseError: Failed to parse: Failed to parse: localhost:3141:

Adding "user data" to an index

We would like to propose to being able to store user data along with an index (field user_data_ stored as json.

In our particular case, this would be used to tell our toolchain what to do when attempting to upload or push from one index to another (we have created a thin wrapper around devpi-client).

For instance, devpi currently has built-hooks for jenkins but as we are using another build management system, this doesn't apply (all this based on the user data).

Here is a scenario:

  • Anne works on project A which has been structured with a user index anne/A, A/dev and A/prod
  • When Anne want to upload a package on her dev index (anne/A), the upload should take place and then a build to verify the package should be trigged. User data attached to index anne/A contains information about the procedure to run.
  • When Anne wants to push from anne/A to A/dev, a build is first triggered. In order to figure out which server and which "job" should be triggered, the devpi wrapper examine the user data attached to the dev index.
  • The build system runs the tests and upon successful completion, pushes to A/dev.
  • If Anne want to push to the A/prod index, she will issue a release request. This time however, the build system realizes that it must trigger a workflow (approval) before the final push takes place.

documentation not pushed along the release file

When performing an upload to index the documentation gets uploaded (--with-docs).

However, when pushing the release file from one index to another, the documentation is not pushed (and I assumed the test results aren't either).

How to reproduce:

  • Upload a release file (with docs) to index A
  • Push the release file to index B
  • Attempt to access the documentation via the web server.

[devpi upload] with-docs option : support for --from-doc-dir

$ devpi --version
0.9.4

Currently the --with-docs uses the setup.py upload_docs command. The setuptools upload_docs will attempt to build any documentation, and then upload it from build/docs. This works well with the default file layout but doesn't work otherwise.

setuptools upload_docs does support the --upload-dir enabling the user to specify another location where the generated documentation can be found.

The following would be useful additions to devpi-client:

  1. Add the --from-doc-dir option allowing the end user to override the setuptools default location.
  2. Being able to define a relative path (to the current project) to the built documentation into a devpi configuration file (e.g. default_docs = ./docs/html) which would be applied by default.

Regarding 2), our organization generates the documentation at the same location for every project therefore this would be a useful configuration option to have.

devpi has hidden dependency to pip>=1.4

Hi,

While deploying a server via --gendeploy, devpi-server raises assertions due to unknown command line option --pre, used in the underlying pip command.

#!sh
$ devpi-server --gendeploy=/opt/devpi_server --serverdir /opt/devpi_server_data --outside-url myurl.domain.net
creating virtualenv: /opt/devpi_server
installing devpi-server,supervisor,eventlet into virtualenv

Usage:   
  pip install [options] <requirement specifier> ...
  pip install [options] -r <requirements file> ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --pre
Traceback (most recent call last):
  File "/usr/bin/devpi-server", line 9, in <module>
    load_entry_point('devpi-server==1.0', 'console_scripts', 'devpi-server')()
  File "/usr/lib/python2.6/site-packages/devpi_server/main.py", line 32, in main
    return gendeploy(config)
  File "/usr/lib/python2.6/site-packages/devpi_server/gendeploy.py", line 156, in gendeploy
    "supervisor", "eventlet", "devpi-server>=%s" % version])
  File "/usr/lib/python2.6/site-packages/devpi_server/gendeploy.py", line 160, in subproc
    return subprocess.check_call([str(x) for x in args])
  File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/devpi_server/bin/pip', 'install', '--pre', '-q', 'supervisor', 'eventlet', 'devpi-server>=1.0']' returned non-zero exit status 2

$ pip --version
pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6)

$ devpi --version
1.0

The problem is that the --pre option was introduced in 1.4 (2013-07-23) but this dependecy is not reflected while installing devpi package. Actually, I would expect that the devpi package's dependencies should express this (pip>=1.4) and the installation on systems with an older version of pip (as in my case) fails.

Also, reading the description of --pre in pip manual

--pre

Include pre-release and development versions.
By default, pip only finds stable versions.

I wonder if adding --pre as a default action to the underlying pip command is a sane default behaviour.

continue uploading further packages using --fromdir if a package 'excepts' for some reason

During an upload --fromdir somedir the process would stop no rst2pdf, as this package has no PKG_INFO file. It would stop on files without name in the pkg_info return value ( reportlab 2.7) and on .egg files. Al these has been resolved with this merge (dd 20130621), but any other causes for the retrieval of pkg_info to fail, will result in aborted upload attempt.

A commandline option could allow the user to ignore such 'excepting' archives, thus allowing the upload of directories with such archives, without having to remove the individual offending archives first.

The message if port 3141 is taken and the automatic server cannot be started should be more clear

If you already have devpi-server running on port 3141 (in non-automatic mode) and issue:

   devpi install --venv=v1 packagename

you get the error message:

   need to be using a live index, see 'devpi use'

This is not a very clear indication the automatic server could not be started.

Document SNI compatible mode on Python 2.x

devpi-client uses the requests library which in turn uses the vendored urllib3 library.

Given Python 2.x's lack of SNI support urllib3 added support for it based on a few packages which when available add the feature even on 2.x. The actually packages to install are only documented in the file though: https://github.com/shazow/urllib3/blob/97f0c840052c696490be1e3cc77e00bc30f78b44/urllib3/contrib/pyopenssl.py#L5-L7

So I think it would be worthwhile to add this to a section of devpi somewhere:

  • pyOpenSSL (tested with 0.13)
  • ndg-httpsclient (tested with 0.3.2)
  • pyasn1 (tested with 0.1.6)

Add option to require authentication for read access

It seems as though right now authentication can only be made a requirement for uploads. I'd like to be able to set authentication for simple download of packages.

Doing so would allow us to have a devpi-server running in the same cloud as our production boxes for quicker deploys, while also being able to publish private packages to it. As I understand the code right now, the only way to protect those private assets would be to keep the server behind a firewall only accessible by the production servers, and that would make publishing to devpi-server from outside that network more difficult.

devpi index --delete fails (typo in the code)

There is a typo in the devpi/index.py

#!python

def main(hub, args):
    hub.requires_login()
    indexname = args.indexname
    if args.delete:
        if not indexname:
            hub.fatal("need to specify index for deletion")
        if arg.keyvalues:
            hub.fatal("cannot --delete if you specify key=values")
        return index_delete(hub, indexname)

Where if arg.keyvalues: should be if args.keyvalues:

As a result we can not delete an index (hence major).

devpi use -l should take an user as an option argument

devpi use -l is very useful. I am just thinking that on a large scale deployment the output might be overwhelming.

I think that people could be interested in seeing the index for a particular user if they want to use one of them as a base, therefore, being able to specify that user would be useful (to filter the output) especially for windoze users with limited ability to pipe this through grep for instance.

devpi should not allow to delete the root user

Version: from repository
changeset: 269:6ea839f0f76a user: holger krekel

It is currently possible to delete the root user which should not happen.

#!bash
$ devpi user -l
200: OK
root
emilie

$ devpi user --delete root
200: user 'root' deleted

$ devpi user -l
200: OK
emilie

I am expecting a 403 HTTP error code (Forbidden). I will add a test which will verify this.

index bases clean up (dangling)

If a base is deleted, this is not reflected in the other indices using this base.

While I don't see any "adverse" effect as in issue #32, this might appear to be a little untidy.

If an index inherits from a base say /user1/dev and that index is deleted, it still shows in the list of bases as illustrated below

#!python

       "user1": {
            "indexes": {}, 
            "username": "user1"
        }, 
        "user2": {
            "indexes": {
                "dev": {
                    "acl_upload": [], 
                    "bases": [
                        "user1/prod"
                    ], 
                    "type": "stage", 
                    "uploadtrigger_jenkins": null, 
                    "volatile": true
                }
            }, 
            "username": "user2"
        }

devpi upload ignores virtualenv?

I am trying to upload packages to my devpi package index but I can't get around uploading my custom Django branch.

First I called "devpi use --venv=/path/to/venv"

Then I change to my Django directory and call "devpi upload" and get this:

vagrant@precise64:/shared/django$ devpi upload
created workdir /tmp/devpi7
hg-exported project to <Exported /shared/django>
--> $ /usr/bin/python /usr/local/lib/python2.7/dist-packages/devpi/upload/setuppy.py /shared/django http://localhost:3141/root/dev/ test test register -r devpi
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/devpi/upload/setuppy.py", line 37, in <module>
    run_setuppy()
  File "/usr/local/lib/python2.7/dist-packages/devpi/upload/setuppy.py", line 31, in run_setuppy
    execfile("setup.py", namespace)
  File "setup.py", line 84, in <module>
    version = __import__('django').get_version()
ImportError: No module named django
Traceback (most recent call last):
  File "/usr/local/bin/devpi", line 9, in <module>
    load_entry_point('devpi-client==0.9.3', 'console_scripts', 'devpi')()
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 24, in main
    return method(hub, hub.args)
  File "/usr/local/lib/python2.7/dist-packages/devpi/upload/upload.py", line 41, in main
    exported.setup_register()
  File "/usr/local/lib/python2.7/dist-packages/devpi/upload/upload.py", line 278, in setup_register
    cwd = self.rootpath)
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 216, in popen_output
    return check_output(args, cwd=str(cwd))
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 45, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/bin/python', '/usr/local/lib/python2.7/dist-packages/devpi/upload/setuppy.py', '/shared/django', 'http://localhost:3141/root/dev/', 'test', 'test', 'register', '-r', 'devpi']' returned non-zero exit status 1

Why is devpi upload trying to use my system's default Python, not one from virtualenv? Did I not instruct it properly? Probably I misunderstand some concepts of package index.

Internal Server Error (500)

Hey,

We have devpi_server 0.9.1 installed and we've seen the following error occur:

Error: 500 Internal Server Error

Sorry, the requested URL 'http://mirror.rcs.griffith.edu.au:3143/root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip%3A' caused an error:

Unhandled exception

The requested URL was:

http://mirror.rcs.griffith.edu.au:3143/root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip:

Here are the relevant parts of the server logs (no tracebacks):


XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:19:06] "GET /root/pypi/+simple/plone.formwidget.autocomplete/ HTTP/1.1" 200 2703
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:19:06] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip HTTP/1.1" 500 877
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:09] "GET /root/pypi/+simple/plone.formwidget.autocomplete/ HTTP/1.1" 200 2703
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:10] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip HTTP/1.1" 200 50169
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:11] "GET /root/pypi/+simple/zc.relation/ HTTP/1.1" 200 319
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:12] "GET /root/pypi/f/https/pypi.python.org/packages/source/z/zc.relation/zc.relation-1.0.tar.gz HTTP/1.1" 200 114754
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:12] "GET /root/pypi/+simple/z3c.objpath/ HTTP/1.1" 200 489
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:13] "GET /root/pypi/f/https/pypi.python.org/packages/source/z/z3c.objpath/z3c.objpath-1.1.zip HTTP/1.1" 200 11491
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:14] "GET /root/pypi/+simple/zope.app.intid/ HTTP/1.1" 200 1665
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:15] "GET /root/pypi/f/https/pypi.python.org/packages/source/z/zope.app.intid/zope.app.intid-3.7.1.tar.gz HTTP/1.1" 200 8161
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:15] "GET /root/pypi/+simple/BTrees/ HTTP/1.1" 200 8236
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:17] "GET /root/pypi/f/https/pypi.python.org/packages/source/B/BTrees/BTrees-4.0.5.tar.gz HTTP/1.1" 200 605375
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:40] "GET /root/pypi/+simple/ipython/ HTTP/1.1" 200 4146
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:44] "GET /root/pypi/f/https/pypi.python.org/packages/2.7/i/ipython/ipython-0.13.1-py2.7.egg HTTP/1.1" 200 3351107
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:46] "GET /root/pypi/+simple/supervisor/ HTTP/1.1" 200 1563
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:47] "GET /root/pypi/f/https/pypi.python.org/packages/source/s/supervisor/supervisor-3.0b1.tar.gz HTTP/1.1" 200 452676
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:48] "GET /root/pypi/+simple/meld3/ HTTP/1.1" 200 933
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:48] "GET /root/pypi/f/https/pypi.python.org/packages/source/m/meld3/meld3-0.6.10.tar.gz HTTP/1.1" 200 41408
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:50] "GET /root/pypi/+simple/z3c.coverage/ HTTP/1.1" 200 1957
XXXX.workstation.griffith.edu.au - - [17/Jun/2013 11:31:50] "GET /root/pypi/f/https/pypi.python.org/packages/source/z/z3c.coverage/z3c.coverage-1.2.0.tar.gz HTTP/1.1" 200 35175
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:36:15] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 880
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:36:15] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:36:29] "GET /root/pypi/+simple/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 890
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:36:29] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:19] "GET /root/pypi/+simple/devpi/ HTTP/1.1" 200 451
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:21] "GET /root/pypi/+simple/devpi-server/ HTTP/1.1" 200 1760
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:22] "GET /root/pypi/f/https/pypi.python.org/packages/source/d/devpi-server/devpi-server-0.9.1.tar.gz HTTP/1.1" 200 46570
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:23] "GET /root/pypi/+simple/py/ HTTP/1.1" 200 5134
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:24] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/py/py-1.4.14.tar.gz HTTP/1.1" 200 188722
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:24] "GET /root/pypi/+simple/execnet/ HTTP/1.1" 200 2841
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:25] "GET /root/pypi/f/https/pypi.python.org/packages/source/e/execnet/execnet-1.1.zip HTTP/1.1" 200 324708
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:26] "GET /root/pypi/+simple/requests/ HTTP/1.1" 200 13354
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:26] "GET /root/pypi/f/https/pypi.python.org/packages/source/r/requests/requests-1.2.3.tar.gz HTTP/1.1" 200 348854
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:29] "GET /root/pypi/+simple/itsdangerous/ HTTP/1.1" 200 2679
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:29] "GET /root/pypi/f/https/pypi.python.org/packages/source/i/itsdangerous/itsdangerous-0.21.tar.gz HTTP/1.1" 200 8595
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:29] "GET /root/pypi/+simple/docutils/ HTTP/1.1" 200 1685
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:29] "GET /root/pypi/f/https/pypi.python.org/packages/source/d/docutils/docutils-0.10.tar.gz HTTP/1.1" 200 1602552
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:31] "GET /root/pypi/+simple/pygments/ HTTP/1.1" 200 15708
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:32] "GET /root/pypi/f/https/pypi.python.org/packages/source/P/Pygments/Pygments-1.6.tar.gz HTTP/1.1" 200 1423161
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:33] "GET /root/pypi/+simple/bottle/ HTTP/1.1" 200 8926
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:41:33] "GET /root/pypi/f/https/pypi.python.org/packages/source/b/bottle/bottle-0.11.6.tar.gz HTTP/1.1" 200 60612
Bottle v0.11.6 server starting up (using WSGIRefServer())...
Listening on http://mirror.rcs.griffith.edu.au:3143/
Hit Ctrl-C to quit.

XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:47:50] "GET /root/pypi/+simple/ HTTP/1.1" 200 21661
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:47:57] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 880
/www/devpi/lib/python2.6/site-packages/devpi_server/views.py:58: DeprecationWarning: Call signature changed (for the better)
  {"content-type": "application/json"})
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:47:57] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:47:59] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 880
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:47:59] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:00] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 880
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:00] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:08] "GET /root/pypi/+simple/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 890
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:08] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:09] "GET /root/pypi/+simple/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 890
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:09] "GET /favicon.ico HTTP/1.1" 404 71
XXXX.staff.ad.griffith.edu.au - - [17/Jun/2013 11:48:10] "GET /root/pypi/f/https/pypi.python.org/packages/source/p/plone.formwidget.autocomplete/plone.formwidget.autocomplete-1.2.4.zip: HTTP/1.1" 500 880

devpi index - unhandled exception when no index name is specified

#!python

$ devpi --version
1.0rc1
$ devpi index
Traceback (most recent call last):
  File "/home/lpbrac/bitbucket/devpi_doc_contrib_1_0/bin/devpi", line 8, in <module>
    load_entry_point('devpi-client==1.0rc1', 'console_scripts', 'devpi')()
  File "/home/lpbrac/bitbucket/devpi_doc_contrib_1_0/local/lib/python2.7/site-packages/devpi/main.py", line 34, in main
    return method(hub, hub.args)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib_1_0/local/lib/python2.7/site-packages/devpi/index.py", line 89, in main
    return index_show(hub, indexname)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib_1_0/local/lib/python2.7/site-packages/devpi/index.py", line 53, in index_show
    ixconfig = get_indexconfig_reply(hub, indexname, ok404=False)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib_1_0/local/lib/python2.7/site-packages/devpi/index.py", line 41, in get_indexconfig_reply
    url = hub.current.get_index_url(indexname, slash=False)
  File "/home/lpbrac/bitbucket/devpi_doc_contrib_1_0/local/lib/python2.7/site-packages/devpi/use.py", line 146, in get_index_url
    raise ValueError("no index name")
ValueError: no index name

devpi use: endpoint without root url results in an exception

version: 0.9.4

The url positional argument of devpi use specifies:

url set current API endpoints to the ones obtained from the
given url.** If already connected to a server**, you can
specify '/USER/INDEXNAME' which will use the same server
context. If you specify the root url you will not be
connected to a particular index.

However attempting to use an existing index without the root url results in an exception (note the "killed automatic server" below)

#!python

$ devpi use /laurent/prod
killed automatic server pid=12429
Traceback (most recent call last):
  File "/usr/local/bin/devpi", line 9, in <module>
    load_entry_point('devpi-client==0.9.4', 'console_scripts', 'devpi')()
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 24, in main
    return method(hub, hub.args)
  File "/usr/local/lib/python2.7/dist-packages/devpi/use.py", line 123, in main
    current.configure_fromurl(hub, args.url)
  File "/usr/local/lib/python2.7/dist-packages/devpi/use.py", line 81, in configure_fromurl
    data = hub.http_api("get", url.rstrip("/") + "/+api", quiet=True)
  File "/usr/local/lib/python2.7/dist-packages/devpi/main.py", line 95, in http_api
    r = methodexec(url, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 347, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3141): Max retries exceeded with url: /laurent/prod/+api (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

However, using the getjson sub-command works nicely (and show the index exists):

#!python

$ devpi getjson /laurent/prod
{
    "result": {
        "bases": [
            "/root/pypi"
        ], 
        "type": "stage", 
        "volatile": false
    }, 
    "status": 200, 
    "type": "indexconfig"
}

note: devpi use http://localhost:3141 (root url) was already issued.

devpi fails to get packages from pypi.python.org

I cannot use my index root/dev which inherits from root/pypi to install packages which are provided in root/pypi because devpi fails to fetch from pypi.python.org although if you install with pip without using devpi-hosted index then pypi.python.org works. Here is what i see in logs:

2013-08-25 11:39:37,932 [INFO ] devpi_server.extpypi: slow access (big reply) to full extpypi simple list
2013-08-25 11:39:38,523 [INFO ] requests.packages.urllib3.connectionpool: Starting new HTTPS connection (14): pypi.python.org
Traceback (most recent call last):
  File "/home/ubuntu/devpi-venv/bin/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/home/ubuntu/devpi-venv/bin/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/home/ubuntu/devpi-venv/local/lib/python2.7/site-packages/devpi_server/views.py", line 590, in indexroot
    for entry in stage.getreleaselinks(name):
TypeError: 'int' object is not iterable

Wheel upload via client not enabled

The changelog is a little conflicted. It says:

  • initial wheel support: "devpi upload --format=bdist_wheel" now uploads
    a wheel format file to the index. (XXX "devpi install" will trigger
    pip commands with option "--use-wheels".)

However, in devpi / client / devpi / install.py, line 36 there is:
#"--use-wheel",

Should this be enabled?

devpi client could provide more obvious error messages

It would be nice if HTTP return codes could be interpreted on the devpi client side, avoiding referring to their definition.

For instance, when attempting to upload a package when the session is expired we get:

#!bash

$ devpi upload --from-dir dist/
401: could not register archimede0.2.0 to http://localhost:3141/emilie/dev/
401: could not register archimede0.1.0 to http://localhost:3141/emilie/dev/

Having the Unauthorized next to the return code would tell the user they have to look in the authentication direction (login/password/user creation)

provide option for upload that only uploads latest version if multiple packages are found

If under the path --fromdir there are multiple versions of a package, allow selection of only the 'latest' version to be uploaded.

This could be implemented by splitting the loop in main_fromdir() in upload.py to first gather all pkg_info (as far as possible), and creating a archivepath to pkg_info dictionary.
If the option for only latest is selected this dictionary should be reduced based on pkg_info.name and .version.
After that for each element in the dictionary upload_file_pypi() should be called.

Document inheritance semantics (including "double" bases)

Version: From repos

  • changeset: 295:28c7f116002c holger krekel
  • summary: bump version numbers (and upload to hpk/dev index)

When creating (by accident) an index which has more than one base, and one of the base is the base of one in the list, both are added (multiple paths)

For instance, we have

#!bash

  /root/pypi
       ^
       |
  /root/dev

but when issuing

#!python

$ devpi index -c /emilie/funky bases=/root/dev,/root/pypi
/emilie/funky:
  type=stage
  bases=root/dev,root/pypi
  volatile=True
  uploadtrigger_jenkins=None
  acl_upload=

we end up with both bases specified (hence it would look like this)

#!bash

  /root/pypi
       ^
       +--------\
       |        |
  /root/dev     |
       ^        |
       |        |
     /emilie/funky

file and directory permissions on the data directory

If you want to use 'devpi upload --fromdir ' to copy all the 'cached' archives from your workstation to your laptop, as the permissions on the files are 600.
Only the toplevel directory ~/.devpi should have limited permissions so that it is easy to copy material if a user decides to put the data somewhere on an exported drive.

Some packages are not installable

Tested on devpi-server 0.9.2

#!bash

$ pip install python-memcached==1.48
Downloading/unpacking python-memcached==1.48
  Could not find any downloads that satisfy the requirement python-memcached==1.48
No distributions at all found for python-memcached==1.48

And the traceback:

127.0.0.1 - - [03/Jul/2013 17:10:21] "GET /root/pypi/+simple/python-memcached/1.48 HTTP/1.0" 500 780
2013-07-03 17:10:21,502 [INFO ] devpi_server.extpypi: visiting crawlurl <DistURL url=u'ftp://ftp.tummy.com/pub/python-memcached/'>
2013-07-03 17:10:21,503 [INFO ] devpi_server.extpypi: crawlurl <DistURL url=u'ftp://ftp.tummy.com/pub/python-memcached/'> <devpi_server.main.FatalResponse instance at 0x7f45287c6998>
2013-07-03 17:10:21,503 [WARNI] devpi_server.extpypi: crawlurl <DistURL url=u'ftp://ftp.tummy.com/pub/python-memcached/'> status <devpi_server.main.FatalResponse instance at 0x7f45287c6998>
2013-07-03 17:10:21,503 [INFO ] devpi_server.extpypi: visiting crawlurl <DistURL url=u'ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz'>
2013-07-03 17:10:21,504 [INFO ] devpi_server.extpypi: crawlurl <DistURL url=u'ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz'> <devpi_server.main.FatalResponse instance at 0x7f45287c6d40>
2013-07-03 17:10:21,504 [WARNI] devpi_server.extpypi: crawlurl <DistURL url=u'ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz'> status <devpi_server.main.FatalResponse instance at 0x7f45287c6d40>
2013-07-03 17:10:21,504 [INFO ] devpi_server.extpypi: visiting crawlurl <DistURL url=u'http://www.tummy.com/Community/software/python-memcached/'>
2013-07-03 17:10:21,999 [INFO ] devpi_server.extpypi: crawlurl <DistURL url=u'http://www.tummy.com/Community/software/python-memcached/'> <Response [200]>
Traceback (most recent call last):
  File "/home/devpi/devpi/bin/bottle.py", line 764, in _handle
    return route.call(**args)
  File "/home/devpi/devpi/bin/bottle.py", line 1575, in wrapper
    rv = callback(*a, **ka)
  File "/home/devpi/devpi/lib/python2.7/site-packages/devpi_server/views.py", line 162, in simple_list_project
    result = stage.getreleaselinks(projectname)
  File "/home/devpi/devpi/lib/python2.7/site-packages/devpi_server/extpypi.py", line 208, in getreleaselinks
    for link in releaselinks]
  File "/home/devpi/devpi/lib/python2.7/site-packages/devpi_server/filestore.py", line 25, in maplink
    key = self.keyfs.PYPIFILES(relpath=link.torelpath())
  File "/home/devpi/devpi/lib/python2.7/site-packages/devpi_server/urlutil.py", line 131, in torelpath
    assert parsed.scheme in ("http", "https")
AssertionError

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.