`devpi use` fails with 401 Unauthorized for Basic auth (6.x regression) about devpi

[fschulze]

Which versions did you try? devpi-client 6.0.4 has some changes which either broke this or fixed it compared to previous 6.x versions.

I strongly recommend looking into devpi-lockdown and devpi-tokens instead of using basic auth.

[fschulze]

Since 6.x credentials are stored in ~/.devpi/client/auth.json.

[vytas7]

I tried debugging Git main a bit, and 6.0.4 from PyPi.
I'll try earlier 6.0.x and report. IIRC most of the things actually work, it is that validation of index that doesn't inherit credentials and fails.

[vytas7]

Hi again, I tried 6.0.0, 6.0.1, 6.0.2, and 6.0.3, all exhibit the same problem as 6.0.4 / main.
5.2.3 works as expected.

Devpi-lockdown might work for us, but it looked a bit suboptimal when we checked it last. Currently, we use Devpi internally without any basic auth or lockdown (but still have users to upload of course), and we have another location in Nginx with TLS cert and basic auth, and IP lock to a list of addresses used by our CI provider. We upload packages to the CI index from external CI (BitBucket Pipelines... 😬).

[vytas7]

So, to summarize, what is happening (on main with a couple of cosmetic debug lines and changing http_api() to raise):

devpi use https://username:[email protected]/ci/ci

<...>
http_api(method='get', url=URL('https://devpi.example.com/ci/ci/+api'), auth is set: True
<...>
http_api(method='get', url=URL('https://devpi.example.com/ci/ci?no_projects='), auth is set: False
Traceback (most recent call last):
  File "/home/vytas/.virtualenvs/devpi/bin/devpi", line 8, in <module>
    sys.exit(main())
  File "/home/vytas/progs/devpi/client/devpi/main.py", line 46, in main
    return method(hub, hub.args)
  File "/home/vytas/progs/devpi/client/devpi/use.py", line 541, in main
    hub.validate_index_access()
  File "/home/vytas/progs/devpi/client/devpi/main.py", line 331, in validate_index_access
    reply = self.http_api(
  File "/home/vytas/progs/devpi/client/devpi/main.py", line 208, in http_api
    raise RuntimeError(f'{r.status_code=} {r.reason=}')
RuntimeError: r.status_code=401 r.reason='Unauthorized'

debugging changes:

--- a/client/devpi/main.py
+++ b/client/devpi/main.py
@@ -150,6 +150,7 @@ class Hub:
         If type is specified and the json result type does not match,
         bail out fatally (unless fatal = False)
         """
+        print(f'http_api({method=}, {url=}, auth is set: {auth is not notset}')
         assert kvdict is None or isinstance(kvdict, (dict, list))
         if isinstance(url, URL):
             url = url.url
@@ -204,6 +205,7 @@ class Hub:
             return reply
         # feedback reply info to user, possibly bailing out
         if r.status_code >= 400:
+            raise RuntimeError(f'{r.status_code=} {r.reason=}')
             if fatal:
                 out = self.fatal
             elif quiet:

FWIW, @fschulze you're right that the basic auth credentials are correctly stored in ~/.devpi/client/auth.json, I missed that. So as said, everything seems to work except that it spits out an error at the end when trying to validate access without actually supplying that basic auth. The same problem with devpi login if you source the DEVP_INDEX envvar, for instance.

[fschulze]

You should add another debug output before

devpi/client/devpi/main.py

Lines 170 to 171 in d4ab0f5

	r = self.http.request(method, url, data=data, headers=headers,
	auth=basic_auth, cert=cert, verify=verify)

, as the call for checking the index access intentionally has no auth set and should do the lookup on

devpi/client/devpi/main.py

Lines 166 to 167 in d4ab0f5

	if basic_auth is notset:
	basic_auth = self.current.get_basic_auth(url=url)

For your use-case you would use devpi-tokens for your CI provider. You can even limit the token permissions to only allow uploads of specific packages. You could get rid of the certificate and basic auth and use devpi-lockdown only for the nginx facing your CI. You would keep the IP limiting setup there. Whether devpi-lockdown is in use is solely determined by the nginx config. So for your internal access you would use an nginx config which doesn't enable devpi-lockdown.

[vytas7]

Hi again,
devpi-lockdown + devpi-tokens sounds promising, but there is very little documentation, and/or probably we are still misunderstanding some concepts.

With basic auth as it is now, we configure pip to use https://username:[email protected]/some/index/+simple/ via pip.conf or otherwise to install packages, run tox, etc.
How does one configure pip to authenticate with devpi-lockdown or to pass the appropriate token?

[fschulze]

I should add examples to the readme of devpi-tokens.

1. devpi token-create -o citoken -a pkg_read,toxresult_upload,upload with optionally -i and -p to further limit to an index and/or a project. This creates a token for the current user that is limited to upload and install. With only pkg_read the token is read-only and only allows installs.
2. then use https://:[email protected]/some/index where token is the content of the created citoken file. The leading : makes sure the token is treated as a password in pip output (if the empty user name before the colon doesn't work for some reason, use __token__ or the original user name used to create the token). It is best to set the PIP_INDEX_URL environment variable instead of the -i/--index-url option, as pip will hide the password in output and it is much less likely to show up in log output.

With token-derive you can take an existing token and limit it further. This is used automatically when token-login is in use to create a one time token for each release when using devpi upload. It could also be useful to create a new token with a shorter expiration (about the time the CI run takes plus some margin) at the start of the CI run, which is then used for pip etc. If it unintentionally shows up in log output, it is less useful, as it will most likely already be expired.

[vytas7]

Thanks, that helps, we'll look into that!
In the short term we could keep pinning devpi-client to 5, but of course it would be nice to resolve the bug as well.

It seems that maybe there's a misunderstanding in the port :443 creeping in somehow resulting in a different key in the url-lookup?
I've now added more debug:

The password is stored unencrypted!
http_api(method='get', url=URL('https://devpi.example.com/ci/ci/+api'), auth is set: True
Use of basic authentication is deprecated, take a look at devpi-lockdown instead. If that doesn't work for you, let us know by filing an issue with details of your usecase.
_get_basic_auth_dict(): self._basic_auth={'https://devpi.example.com:443/': ['pipelines', '***']}
<...>
supported features: server-keyvalue-parsing
http_api(method='get', url=URL('https://devpi.example.com/ci/ci?no_projects='), auth is set: False
_get_basic_auth_dict(): self._basic_auth={'https://devpi.example.com:443/': ('pipelines', '***')}
current.get_basic_auth(url='https://devpi.example.com/ci/ci?no_projects=') ==> None
<...>
RuntimeError: r.status_code=401 r.reason='Unauthorized'

[henryborchers]

I don't know if this is related but I noticed that on Python 3.11 only for on Linux, the value from auth.json failed because it does not properly escaping quotes.

I can log into the devpi index but if I run "devpi test" with "-e py311" I get the following stack trace...

ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 375, in run
    self.trace_basic_info(finder)
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 475, in trace_basic_info
    locations = search_scope.get_formatted_locations()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/models/search_scope.py", line 82, in get_formatted_locations
    redacted_index_url = redact_auth_from_url(url)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/utils/misc.py", line 527, in redact_auth_from_url
    return _transform_url(url, _redact_netloc)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/devpi-test-nnanobpf/targz/HathiZip-0.1.11.dev0/.tox/py311/lib/python3.11/site-packages/pip/_internal/utils/misc.py", line 492, in _transform_url
    purl = urllib.parse.urlsplit(url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/parse.py", line 500, in urlsplit
    _check_bracketed_host(bracketed_host)
  File "/usr/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
    ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')

This doesn't happen with any other version of Python or if I use another os.

update: formatting

[fschulze]

@vytas7 since you are using basic auth, you must have a proxy in front of devpi-server. If that is nginx, compare your config with the current example created with a current version of the devpi-gen-config script. There have been some fixes related to $scheme which might have an effect. Also see #840 But I have a feeling this is not the cause of your issues with devpi-client 6.x.

[henryborchers]

@fschulze the thing that I don't understand is why is this only happening when running devpi test with python 3.11 only

[henryborchers]

I don't think my issue is the same. So I created a new ticket #978

[fschulze]

@henryborchers there recently were some urllib/urlparse changes/security fixes which I think could cause that and were introduced first in 3.11 and only just a few days ago backported to older Python versions, so this might show up with those as well soon.

[fschulze]

By now I'm pretty sure this is a quoting problem. Could you all try to use urllib.parse.quote on your password and try the result instead of your real password? I suspect that there are characters in your password which cause this.

For the basic auth, my guess is that the passing of the URL to pip via the environment variable doesn't quote the password correctly.

For Python 3.11, the changes in urlsplit most likely trip over the unquoted password.

[vytas7]

@fschulze thanks, I'll check the Nginx config. 3.11 is unrelated to our problems since we're mostly on 3.8.

[fschulze]

Could you try:

https://m.devpi.net/fschulze/dev/devpi-client/6.0.5.dev0
https://m.devpi.net/fschulze/dev/+f/8e0/51c0ad3c869d5/devpi_client-6.0.5.dev0-py2.py3-none-any.whl