devpi can install package that is not in mirror_whitelist about devpi

[fschulze]

The documentation at https://devpi.net/docs/devpi/devpi/stable/+doc/userman/devpi_indices.html?highlight=mirror_whitelist#modifying-the-mirror-whitelist is a bit vague.

By default a locally uploaded package will prevent releases from mirrors to be seen. That is to prevent an attack where someone uploads a malicious package on PyPI which targets a private package. With mirror_whitelist one can disable that for packages that you develop yourself, or for which you upload custom built wheels etc.

For your use-case, this may be better: https://pypi.org/project/devpi-constrained/

[BenjaminYde]

Thanks for the quick response!

I made my use-case work with devpi-constrained:
When setting constraints=mypacakge-1,mypackage2,* i was able to make sure only package 1 & 2 could be installed.

Follow-up Questions:

• Is it possible that mirror & mirror_whitelists only works with public mirrors like pypi.org?
• Is it possible to mirror local devpi package indexes? (when using devpi-lockdown)

[fschulze]

* Is it possible that mirror & mirror_whitelists only works with public mirrors like pypi.org?

It only applies to type=mirror indexes in bases of the index mirror_whitelist is set on.

* Is it possible to mirror local devpi package indexes? (when using devpi-lockdown)

Not sure what you mean. If you mean mirroring from a separate devpi-server instance, then yes, that works in general. If the devpi-server instance you want to mirror also has devpi-lockdown, you would have to provide credentials in the mirror_url. I would recommend devpi-tokens also see #965 (comment)

[BenjaminYde]

That is clear to me now, thanks.

Request: define all constraints in 1 command

With devpi-constraints i came across the following problem:

devpi index constraints="package-1,package-2,*"

/client-a/project-c constraints=package-1,package-2,*
PATCH http://devpi:3141/client-a/project-c
400 Bad Request: Error while parsing constrains: Parse error at "',package'": Expected stringEnd

User story: I want to be able to define all constraints using 1 command without a constraints.txt
constraints are: package-1,package-2,*

The following commands should be able to work:
devpi index constraints=package-1,package-2,*
devpi index constraints="package-1,package-2,*"

The following works:

Option 1:
devpi index constraints=package-1
devpi index constraints+=package-2
devpi index constraints+="*"

Option 2:
devpi index constraints="$(cat ./constraints.txt)"

The following does not work:

devpi index constraints=package-1,package-2
devpi index constraints="package-1,package-2"

devpi index constraints=package-1,package-2,*
devpi index constraints="package-1,package-2,*"

Another problem i have is that when package-1 or package-2 have dependencies, they cannot be installes because of the '*'
Is there a way i could fix this?

[fschulze]

@BenjaminYde that is because the constraints option is meant to be like a constraints.txt file for pip, which uses newlines. Instead of a comma, use \n or maybe \\n. The problem is, that a naive replacement of commas with newlines doesn't work, because constraints can be something like pkg>1,!=1.2. If you have an idea on how to fix that, please report it in devpi-constraints issue tracker, or make a PR there.

[fschulze]

Another problem i have is that when package-1 or package-2 have dependencies, they cannot be installes because of the '*' Is there a way i could fix this?

I can't think of an easy way besides parsing the dependencies and adding them. Maybe use the output of pip list or pip freeze with some editing/processing.

[BenjaminYde]

I tried to create a new package-index that is of type=mirror and has a mirror_url to a package-index of the same devpi instance. It works with devpi-lockdown. devpi install package-1 works, but i want to limit the option to install other packages.

The problem i have is that when I try todo devpi index mirror_whitelist=package-1 it does not add this property to the index.
Does this behaviour work as intended?

[fschulze]

@BenjaminYde why would you try to mirror from the same instance?

The mirror_whitelist option is only relevant on regular indexes, if it is possible to set it on a mirror index, then that is a bug.

[BenjaminYde]

I am using the type=mirror because i want to use the mirror_whitelist option.
mirror_whitelist enables me to add the following contraint: limit what packages can be installed from that index.
The following use case will explain why it needs to be done from the same instance.

Use case:

We have a monorepo company/mono with 20 packages on it (obfuscated)
Clients need their own package index (for example client-a/project-a)
I want to create a mirror from company/mono and give it the name client-a/project-a

Clients should not be able to download all the packages, but only a selected few
(only those the project of the client needs)

mirror_whitelist would have helped me achieve that. (but that does not work)
(for example devpi index client-a/project-a mirror_whitelist=package-1)

[fschulze]

For that use case I would say it is better to have one index per package, set bases on each client index, so they see only the projects they should. Then use #965 (comment) to limit access per client, so they can't see the other indexes. Finally use --restrict-modify, so the clients can't modify their own indexes to see other stuff.

Another way would be to push the releases the clients should see from mono to each clients index, but currently that would make copies of the packages on disk. Each upload would also require a push for each client. So overall, the first approach is most likely better.

[BenjaminYde]

I followed all the steps you said (which work! thank you for that!):

• Created an index per package of the mono repo
• Created a client index that inherits only the indexes (per package) that it needs
• Created a token with pkg_read only
• Tested everything with devpi install
• The advantage with devpi-tokens is that the clients can use this token instead of username & password. this avoids them to login to the devpi-web!

I still have 1 question about this setup:
What is the advantage here when you compare the following 2 options:

Option 1: index per package + use inherit in client-a/project-a + use token with explicit index

• Creating a package index for each package of the mono repo
• Reference them in the devpi index bases=company-mono/package-1 in client-a/project-a
• Create a token with only pkg_read access to client-a/project-a

Option 2: 1 index for all packages + use inherit for client-a/project-a + use token with explicit index & projects

• Do not create a package index for each package of the mono repo
• Upload all packages to the index company/mono
• Create an index client-a/project-a with bases=company/mono
• Create a token with
  • --allowed=pkg_read
  • --indexes=client-a/project-a
  • --projects=package-1

It would be easier to only maintain 1 index company/mono and explicitly say what packages can be installed with the use of tokens with pkg_read + --indexes + --projects?

[fschulze]

Option 2 should also work if you manage the tokens and give them to the clients instead of the user/password. You can also use the token as password in the browser (normal username, or try __token__), with that you can test various endpoints to see if they are protected or if there is unintentional information leakage. I guess Option 2 might indeed be easier to manage for your use-case. I'm not sure whether you might have to add company/mono to the indexes of the token, you have to try that.

[BenjaminYde]

in devpi-web i used the following usernames with the generated token as password and i could not login:

• <no-username>
• root
• client-a
• client-a/project-d (this gave me an internal error)
• token
• _token_
• __token__
• ___token___
• <generated-token>
• <token-id> (when i do devpi token-list -u client-a)

I would like to know if there is a way to authenticate because I need to know that the clients cannot view the devpi-web

[BenjaminYde]

Option 2 should also work if you manage the tokens and give them to the clients instead of the user/password. You can also use the token as password in the browser (normal username, or try __token__), with that you can test various endpoints to see if they are protected or if there is unintentional information leakage. I guess Option 2 might indeed be easier to manage for your use-case. I'm not sure whether you might have to add company/mono to the indexes of the token, you have to try that.

I went for option 1, because option 2 is not possible for me because of the need for dependencies when using -p or -i
See this issue: devpi/devpi-tokens#3

[fschulze]

@BenjaminYde I investigated a bit and I found no way to use tokens in a browser. There is no hook in devpi-tokens which implements login via token and using the token in the URL in the browser doesn't work either. Though it seems Firefox doesn't store or send basic auth credentials if the website doesn't ask for it via headers, not sure about other browsers.

It is possible to view html via curl though, it happily provides the basic auth from the URL (or via options) and returns the html. So you need to make sure with curl that there are no information leaks.