epicwink / proxpi Goto Github PK

View Code? Open in Web Editor NEW

111.0 6.0 23.0 300 KB

PyPI caching proxy

License: MIT License

Python 87.14% HTML 10.25% Dockerfile 1.11% Shell 1.50%

pypi index mirror cache python

proxpi's People

Stargazers

Watchers

proxpi's Issues

Can I persist the cache between docker runs with a volume mount

I was hoping I could run this with something like:

Problem definition

docker run -v $HOME/pip-cache:/mountpoint epicwink/proxpi

and then be able to persist the cache between different executions of proxpi. Is this possible?

Current behaviour and workarounds

I had a look by hand and it seems like the cached wheels are in a temporary directory under tmp. I had a look at the readme and searched for volumes but couldn't find this.

Cannot download some package

With a brand-new docker install, I cannot download some package.
The latest version raise a 404 for .metadata

Reproduction

Here's some example with Sphinx==7.0.1:

$ pip install sphinx==7.0.1 --index-url=http://proxpi:5000/index/ --ignore-installed --no-cache
Looking in indexes: http://proxpi:5000/index/
Collecting sphinx==7.0.1
  Obtaining dependency information for sphinx==7.0.1 from http://proxpi:5000/index/sphinx/sphinx-7.0.1-py3-none-any.whl.metadata
  ERROR: HTTP error 404 while getting http://proxpi:5000/index/sphinx/sphinx-7.0.1-py3-none-any.whl.metadata
ERROR: 404 Client Error: Not Found for url: http://proxpi:5000/index/sphinx/sphinx-7.0.1-py3-none-any.whl.metadata

With the original repo it works:

$ pip install sphinx==7.0.1 --index-url=https://pypi.org/simple/
Looking in indexes: https://pypi.org/simple/
...

Also previous versions of Sphinx works:

$ pip install sphinx==7.0.0 --index-url=http://proxpi:5000/index/ --ignore-installed --no-cache
Looking in indexes: http://proxpi:5000/index/
Collecting sphinx==7.0.0
  Downloading http://proxpi.cloud-mercato.lan:5051/index/sphinx/sphinx-7.0.0-py3-none-any.whl (3.0 MB)
...

Expected behaviour

I should be able to install the last package transparently

Actual behaviour

Include current behaviour and output.

Environment

proxpi: latest
Environment: Docker container
Python: 3.10.6
OS: macOS M2 Ventura

Support PEP 700

Support PEP 700, which adds project versions and file size and upload-time, and sets api-version to 1.1.

Problem definition

I'm trying to use proxpi in an offline environment: I'm downloading and caching all pip packages I want to move to my offline environment using a proxpi instance in a online one, then I'm moving a tar.gz with all the files to the offline one and mounting it on the proxpi running on it, but then i can't install any package using the offline one.

Maybe I missing some configuration ? This is how my docker-compose.yaml looks like in my online environment;

services:
  proxpi:
    volumes:
      - '/home/ubuntu/proxpi-cache/:/etc/cached/'
    environment:
      - PROXPI_CACHE_DIR=/etc/cached/
    container_name: proxpi-online
    restart: unless-stopped
    ports:
      - '5000:5000'
    image: epicwink/proxpi:latest

And this one is for my offline one:

services:
  proxpi:
    volumes:
      - '/home/ubuntu/proxpi-cache/:/etc/cached/'
    environment:
      - PROXPI_INDEX_URL=http://localhost:5000/index/
      - PROXPI_CACHE_DIR=/etc/cached/
    container_name: proxpi-online
    restart: unless-stopped
    ports:
      - '5000:5000'
    image: epicwink/proxpi:latest

Current behaviour and workarounds

Can't install the packages in a offline environment after moving the files from one volume to another. I haven't find a workaround yet.

Increase download speed with multi-connection request

It's so faster if change _download_file method to support multi-connection download.

proxpi/src/proxpi/_cache.py

Line 654 in caa09de

def _download_file(self, url: str, path: str):

Problem definition

Download is slow for my network situation. (I believe others can experience it, too.)

Current behaviour and workarounds

Proposed solution (optional)

make _download_file method multi-connection.

this is an example:
http://stackoverflow.com/questions/13973188/ddg#13973531

Document how to use this in typical CI runners

This is great! Could you open a space in the README or other documentation to demonstrate using this in Ci runners like travis, azure pipelines, or github actions which are used in many open source projects? The CI runner documentation about best practices with caching is not always clear: when a cache actually saves CI bandwidth and when it doesn't.

Add pypicloud as alternative

https://github.com/stevearc/pypicloud

currently in "maintenance mode" (stevearc/pypicloud#325)
supports s3
supports caching
supports proxy-mode
stateless (perfect for k8)

Attempt to poll main index for changes

PyPI exposes an API for new/updated packages. This can be used to automatically invalidate cache of packages list and package files list.

If the main index is not PyPI and doesn't expose a similar API, the time-based cache-invalidation should be used

Invalid hash when using Poetry

Thanks for the project. I'm trying to use with poetry, and one of the dependencies get a different sha256 from
the Pypi one, which causes poetry install to fail

Reproduction

Steps to reproduce the behavior:

Create a new poetry project: poetry new test-proxpi
on the project dir, without using proxpi yet, run poetry add tornado==5.1.1
Run proxpi locally, then change add the repo on pyproject.toml:

[[tool.poetry.source]]
name = "proxpi"
url = "http://localhost:5000/index/"
default = true
secondary = false

remove the virtualenv created by poetry: rm -rf `poetry env info -p`
run poetry install again

Expected behaviour

Poetry install should succeed

Actual behaviour

Installation fails:

Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

Package operations: 112 installs, 0 updates, 0 removals

  • Installing pycparser (2.21)
  • Installing pyparsing (3.0.9)
  • Installing six (1.16.0)
  • Installing smmap (5.0.0)
  • Installing wrapt (1.14.1)
  • Installing async-timeout (4.0.2)
  • Installing attrs (21.4.0)
  • Installing certifi (2022.5.18.1)
  • Installing cffi (1.15.0): Installing...
  • Installing charset-normalizer (2.0.12)
  • Installing defusedxml (0.7.1): Installing...
  • Installing deprecated (1.2.13)
  • Installing gitdb (4.0.9): Installing...
  • Installing idna (3.3): Installing...
  • Installing iniconfig (1.1.1): Installing...
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3): Installing...
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0): Installing...
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3): Installing...
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0): Installing...
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3): Installing...
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0): Installing...
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0): Installing...
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0): Installing...
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing deprecated (1.2.13)
  • Installing gitdb (4.0.9): Installing...
  • Installing idna (3.3): Installing...
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing defusedxml (0.7.1)
  • Installing deprecated (1.2.13)
  • Installing gitdb (4.0.9): Installing...
  • Installing idna (3.3): Installing...
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing idna (3.3)
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing idna (3.3)
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing gitdb (4.0.9)
  • Installing idna (3.3)
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing markupsafe (2.1.1)
  • Installing packaging (21.3)
  • Installing pbr (5.9.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing pbr (5.9.0)
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0): Installing...
  • Installing py (1.11.0)
  • Installing charset-normalizer (2.0.12)
  • Installing defusedxml (0.7.1)
  • Installing deprecated (1.2.13)
  • Installing gitdb (4.0.9)
  • Installing idna (3.3)
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1)
  • Installing packaging (21.3)
  • Installing pbr (5.9.0)
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0)
  • Installing cffi (1.15.0)
  • Installing charset-normalizer (2.0.12)
  • Installing defusedxml (0.7.1)
  • Installing deprecated (1.2.13)
  • Installing gitdb (4.0.9)
  • Installing idna (3.3)
  • Installing iniconfig (1.1.1)
  • Installing markupsafe (2.1.1)
  • Installing packaging (21.3)
  • Installing pbr (5.9.0)
  • Installing pluggy (1.0.0)
  • Installing protobuf (3.20.1): Installing...
  • Installing py (1.11.0)
  • Installing py (1.11.0)
  • Installing protobuf (3.20.1)
  • Installing py (1.11.0)
  • Installing pyasn1 (0.4.8)
  • Installing python-dateutil (2.8.2)
  • Installing pytz (2022.1)
  • Installing tomli (2.0.1)
  • Installing tornado (5.1.1): Failed

  RuntimeError

  Invalid hashes (sha256:eefad610df4f966186885efeb9ea73c04914b9edb3e3a26ed9b10a562a4b70b0) for tornado (5.1.1) using archive tornado-5.1.1.tar.gz. Expected one of sha256:0662d28b1ca9f67108c7e3b77afabfb9c7e87bde174fbda78186ecedc2499a9d, sha256:4e5158d97583502a7e2739951553cbd88a72076f152b4b11b64b9a10c4c49409, sha256:732e836008c708de2e89a31cb2fa6c0e5a70cb60492bee6f1ea1047500feaf7f, sha256:8154ec22c450df4e06b35f131adc4f2f3a12ec85981a203301d310abf580500f, sha256:8e9d728c4579682e837c92fdd98036bd5cdefa1da2aaf6acf26947e6dd0c01c5, sha256:d4b3e5329f572f055b587efc57d29bd051589fb5a43ec8898c77a47ec2fa2bbb, sha256:e5f2585afccbff22390cddac29849df463b252b711aa2ce7c5f3f342a5b3b444.

  at ~/.pyenv/versions/3.9.9/lib/python3.9/site-packages/poetry/installation/executor.py:627 in _download_link
      623│                     )
      624│                 )
      625│ 
      626│             if archive_hashes.isdisjoint(hashes):
    → 627│                 raise RuntimeError(
      628│                     "Invalid hashes ({}) for {} using archive {}. Expected one of {}.".format(
      629│                         ", ".join(sorted(archive_hashes)),
      630│                         package,
      631│                         archive_path.name,

  • Installing urllib3 (1.26.9)
  • Installing vine (5.0.0)
  • Installing zipp (3.8.0)

Environment

Poetry: 1.1.13
Python: 3.9.9
OS: Linux min 20.02 5.13.0-51-generic

Support simple JSON API

Use and implement the simple JSON index API PEP 691

Prefer and handle JSON-encoded responses from source indexes
Implement optional (prefer HTML) JSON-encoding responses to clients

flask dependency issue

A requirements.txt file stating the dependency on Flask>=2.0 is missing.

Reproduction

Steps to reproduce the behavior:

Upgrade from a proxpi version 0.1.0 to 1.0.0 (with Flask 1.1.x installed) with pip install --user -U proxpi.
Restart the service
Navigate to the main page
Result: 500 internal server error and TypeError: send_file() got an unexpected keyword argument 'max_age' in the logs

Environment

proxpi: 1.0.0
Python: 3.8+
OS: Ubuntu 20.04
flask: 1.1.x

my 2 cents

First, let me thank you for publishing this useful little gem, your effort is much appreciated.

The culprit here is the API change that Flask did when switching to 2.x. I ran proxpi on top of the Ubuntu-provided python3-flask package, which was fine for proxpi 0.1.0, but now it seems that some keyword parameters have changed and that causes the TypeError.

IMHO the easiest (and clearest) fix would be to provide a requirements.txt file with the necessary dependencies.

BTW: I don not run proxpi in docker, I'm using a simple systemd service file, see below.

# /etc/systemd/system/pipcache.service
[Unit]
Description=proxpi pip cache
After=network.target

[Service]
Environment=FLASK_APP=proxpi.server FLASK_DEBUG=0 PYTHONUNBUFFERED=1
ExecStart=flask run --host=0.0.0.0

[Install]
WantedBy=multi-user.target

Cannot find package when add extra index

A paragraph or two of the bug.

Reproduction

Steps to reproduce the behavior:

Run docker with PROXPI_EXTRA_INDEX_URLS
Install as usual: pip install --index-url http://127.0.0.1:5000/index/ numpy
See error

Expected behaviour

Install successfully from simple pip

Actual behaviour

proxpi  | 2023-10-11 09:23:23 [   ERROR] proxpi: Exception on /index/numpy/ [GET]
proxpi  | Traceback (most recent call last):
proxpi  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2525, in wsgi_app
proxpi  |     response = self.full_dispatch_request()
proxpi  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1822, in full_dispatch_request
proxpi  |     rv = self.handle_user_exception(e)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
proxpi  |     rv = self.dispatch_request()
proxpi  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
proxpi  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/proxpi/server.py", line 172, in list_files
proxpi  |     files = cache.list_files(package_name)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/proxpi/_cache.py", line 810, in list_files
proxpi  |     root_files = self.root_cache.list_files(package_name)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/proxpi/_cache.py", line 462, in list_files
proxpi  |     self._list_files(package_name)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/proxpi/_cache.py", line 416, in _list_files
proxpi  |     response = self.session.get(url, headers=self._headers)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
proxpi  |     return self.request("GET", url, **kwargs)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
proxpi  |     resp = self.send(prep, **send_kwargs)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 695, in send
proxpi  |     adapter = self.get_adapter(url=request.url)
proxpi  |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 792, in get_adapter
proxpi  |     raise InvalidSchema(f"No connection adapters were found for {url!r}")
proxpi  | requests.exceptions.InvalidSchema: No connection adapters were found for '"https:/pypi.org/numpy'

Environment

docker: epicwink/proxpi:latest
envs: PROXPI_EXTRA_INDEX_URLS="https://download.pytorch.org/whl/cu102,https://download.pytorch.org/whl/cu113,https://download.pytorch.org/whl/cu116,https://download.pytorch.org/whl/cu117,https://download.pytorch.org/whl/cu118,https://download.pytorch.org/whl/cpu" since I need extra of pytorch for development and deploy proces.

Support PEP 708

Support PEP 708, which adds meta.tracks and alternate-locations, and sets api-version to 1.2. Implement both as server and client.

ARM Docker image

Would it be possible to get Docker ARM images available ?
Currently, only amd64 is available giving bad perfs (even 502) through emulation.

Cache package file not working as expected

Run proxpi docker with PROXPI_CACHE_DIR environment variable

docker run --env PROXPI_CACHE_DIR=/var/cache/proxpi -p 8080:5000 epicwink/proxpi

output:

 * Serving Flask app "proxpi.server"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2022-05-23 11:32:51,080 [    INFO] proxpi._cache: Listing packages in index 'https://pypi.org/simple/'
2022-05-23 11:32:51,086 [    INFO] werkzeug:  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

install package from the proxpi docker

pip install -i http://localhost:8080/index/ gunicorn

output:

Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: http://localhost:8080/index/


Collecting gunicorn
  Downloading http://localhost:8080/index/gunicorn/gunicorn-20.1.0-py3-none-any.whl (79 kB)
     |████████████████████████████████| 79 kB 125.6 MB/s 
Requirement already satisfied: setuptools>=3.0 in /usr/lib/python3/dist-packages (from gunicorn) (45.2.0)
Installing collected packages: gunicorn
Successfully installed gunicorn-20.1.0
WARNING: You are using pip version 21.1.3; however, version 22.1.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

docker exec

docker exec -it 3a671ff06250 sh
ls -lhtra /var/cache/

output:

total 0      
drwxr-xr-x    2 root     root           6 Jan 16  2020 apk
drwxr-xr-x    1 root     root          19 Feb 26  2020 ..
drwxr-xr-x    1 root     root          18 Feb 26  2020 .
drwxr-xr-x    1 root     root           6 Mar 21  2020 misc

I can't find the cache for the gunicorn package that I have just installed, any idea where is the package?
note: find / -iname *gunicorn* return nothing

long-term use of proxpi on a low-resource server

dear EpicWink, i am considering using your caching proxy proxpi on my field (semi-portable) NAS. the server is solely for my use, and it has a pretty weak CPU and 1.5GB of RAM.

i wonder if your caching proxy is suitable for long-term cache storage in case of internet disruptions. for instance, if i set PROXPI_INDEX_TTL=6570000 (several years) and accumulate cache over a few months (and after numerous power cycles), will the server operate normally? will i be able to use the cached files for up to a year?

i would appreciate any information or recommendations you could provide.
best regards, ~le berouque

EDIT: added remarks about "semi-portable" and "power cycles"

[FR]Support for proxy access to upstream servers

Problem definition

In some environments, services cannot access the Internet directly and need to use proxies. For example, HTTP Proxy/Socks5

Current behaviour and workarounds

Proposed solution (optional)

Passing Config Parameters via Environment Variables. e.g.:

HTTP_PROXY=http://192.168.1.1:8080
SOCKS5_PROXY=192.168.1.1:1080

Support Basic HTTP authentication

pip install supports Basic HTTP authentication: https://pip.pypa.io/en/stable/topics/authentication/#basic-http-authentication

example:

pip install -f https://user:[email protected]:443/pkgs requests

It would be amazing if the proxpi will support it

epicwink / proxpi Goto Github PK

proxpi's People

Stargazers

Watchers

Forkers

proxpi's Issues

Problem definition

Current behaviour and workarounds

Reproduction

Expected behaviour

Actual behaviour

Environment

Problem definition

Current behaviour and workarounds

Problem definition

Current behaviour and workarounds

Proposed solution (optional)

Reproduction

Expected behaviour

Actual behaviour

Environment

Reproduction

Environment

my 2 cents

Reproduction

Expected behaviour

Actual behaviour

Environment

Problem definition

Current behaviour and workarounds

Proposed solution (optional)

Recommend Projects

Recommend Topics

Recommend Org