jupyterlab-black fails to install (packages with hyphens in name cause problems?) about pipgrip

[andrewv99]

pip install -U git+https://github.com/ddelange/pipgrip.git@verbose-crash
Looking in indexes: https://bridge4:9443/repository/rdc-python-all/simple
Collecting git+https://github.com/ddelange/pipgrip.git@verbose-crash
Cloning https://github.com/ddelange/pipgrip.git (to revision verbose-crash) to /tmp/pip-req-build-kamhauj7
Running command git clone -q https://github.com/ddelange/pipgrip.git /tmp/pip-req-build-kamhauj7
Running command git checkout -b verbose-crash --track origin/verbose-crash
Switched to a new branch 'verbose-crash'
Branch 'verbose-crash' set up to track remote branch 'verbose-crash' from 'origin'.
Requirement already satisfied, skipping upgrade: anytree in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (2.8.0)
Requirement already satisfied, skipping upgrade: click>=7 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (7.1.2)
Requirement already satisfied, skipping upgrade: packaging>=17 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (20.4)
Requirement already satisfied, skipping upgrade: pkginfo>=1.4.2 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (1.5.0.1)
Requirement already satisfied, skipping upgrade: setuptools>=38.3 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (41.0.1)
Requirement already satisfied, skipping upgrade: wheel in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (0.34.2)
Requirement already satisfied, skipping upgrade: pip>=7.1.0 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from pipgrip==0.6.1.dev1) (19.1.1)
Requirement already satisfied, skipping upgrade: six>=1.9.0 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from anytree->pipgrip==0.6.1.dev1) (1.15.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/andrewv/p3env/pipgrip/lib/python3.6/site-packages (from packaging>=17->pipgrip==0.6.1.dev1) (2.4.7)
Building wheels for collected packages: pipgrip
Building wheel for pipgrip (setup.py) ... done
Stored in directory: /tmp/pip-ephem-wheel-cache-87o5fns1/wheels/ca/0b/34/63023b90dc2ad20b187e30c05ed7ccc28995b50cea48a10642
Successfully built pipgrip
Installing collected packages: pipgrip
Found existing installation: pipgrip 0.6.0
Uninstalling pipgrip-0.6.0:
Successfully uninstalled pipgrip-0.6.0
Successfully installed pipgrip-0.6.1.dev1
WARNING: You are using pip version 19.1.1, however version 20.2.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

pipgrip --verbose --install tensorflow==2.0.2 2>&1|tee err.log
err.log

[ddelange]

Thanks for the report and the PR! I clearly overlooked wheel filenames not necessarily always being canonicalized (also e.g. Keras_Preprocessing-... in test wheels).

Since the actual cli interaction with pip is mocked out (TODO add tests for _download_wheel with different pip versions) I think for now we can do with a quick fix without having to change the function signature. in any case, the dir lookup should be hyphen/underscore/capitalization agostic as wheel naming conventions might change in the future.
in your pdb above, L255

-    if pkg in whl:
+    if pkg in canonicalize_name(whl):

should do the trick for the quick substring check.

will take a closer look on my laptop tomorrow! really need tests here.

[mdengler]

[add canonicalize_name(whl) instead]

Gotcha; done.

[ddelange]

https://github.com/ddelange/pipgrip/releases/tag/0.3.1 🤙

[ddelange]

tests in #23

[andrewv99]

Hi,

I am using 0.6.0, just released, similar symptom happening there:

INFO: derived: absl-py (>=0.7.0)
Traceback (most recent call last):
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/bin/pipgrip", line 10, in
sys.exit(main())
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/cli.py", line 419, in main
solution = solver.solve()
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 69, in solve
if not self._run():
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 85, in _run
next_package = self._choose_package_version()
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 349, in _choose_package_version
term = self._next_term_to_try()
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 337, in _next_term_to_try
term = min(*unsatisfied, key=_get_min)
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 325, in _get_min
term.package, term.constraint.constraint
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/libs/mixology/package_source.py", line 77, in versions_for
return self._versions_for(package, constraint)
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/package_source.py", line 160, in _versions_for
self.discover_and_add(package.req.str())
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/package_source.py", line 126, in discover_and_add
package, self.index_url, self.extra_index_url, self.cache_dir, self.pre
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/pipper.py", line 365, in discover_dependencies_and_versions
req.str(), index_url, extra_index_url, pre, cache_dir
File "/home/andrewv/p3env/tmp_rdc_python_sample_prj/lib/python3.6/site-packages/pipgrip/pipper.py", line 292, in _download_wheel
str({package: os.path.join(cache_dir, fname.lstrip(os.path.sep))})
UnboundLocalError: local variable 'fname' referenced before assignment

[ddelange]

Hi @andrewv99,

1. for this exact environment, can you paste the output of pip wheel absl-py>=0.7.0?
   I suspect this env has an old version of pip installed (that should still work, but somehow absl-py does not get matched against the wheels in your wheel dir).
2. can you try adding -vvv --no-cache-dir to your pipgrip command and paste the resulting output here if that also fails?
3. can you try your original command again with this branch: pip install -U git+https://github.com/ddelange/pipgrip.git@verbose-crash? it should now crash with a verbose message that'll help debug this :)

[andrewv99]

Hi,

Thanks for the quick response. I have been trying to reproduce it, and I can now reproduce it with pip 19.1.1 and these dependencies installed, but if I replace pip with 20.2.1, the problem does not occur:

pipdeptree
certifi==2020.6.20
networkx==2.3

• decorator [required: >=4.3.0, installed: 4.4.2]
  pip-tools==3.8.0
• click [required: >=6, installed: 7.1.2]
• six [required: Any, installed: 1.15.0]
  pipdeptree==1.0.0
• pip [required: >=6.0.0, installed: 19.1.1]
  pipgrip==0.6.0
• anytree [required: Any, installed: 2.8.0]
  • six [required: >=1.9.0, installed: 1.15.0]
• click [required: >=7, installed: 7.1.2]
• packaging [required: >=17, installed: 20.4]
  • pyparsing [required: >=2.0.2, installed: 2.4.7]
  • six [required: Any, installed: 1.15.0]
• pip [required: >=7.1.0, installed: 19.1.1]
• pkginfo [required: >=1.4.2, installed: 1.5.0.1]
• setuptools [required: >=38.3, installed: 41.0.1]
• wheel [required: Any, installed: 0.34.2]

And if I run:
pipgrip --verbose --install tensorflow==2.0.2

I get:
Traceback (most recent call last):
File "/home/andrewv/p3env/pipgrip/bin/pipgrip", line 8, in
sys.exit(main())
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/cli.py", line 419, in main
solution = solver.solve()
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 69, in solve
if not self._run():
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 85, in _run
next_package = self._choose_package_version()
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 349, in _choose_package _version
term = self._next_term_to_try()
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 337, in _next_term_to_t ry
term = min(*unsatisfied, key=_get_min)
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/libs/mixology/version_solver.py", line 325, in _get_min
term.package, term.constraint.constraint
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/libs/mixology/package_source.py", line 77, in versions_for
return self._versions_for(package, constraint)
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/package_source.py", line 160, in _versions_for
self.discover_and_add(package.req.str())
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/package_source.py", line 126, in discover_and_add
package, self.index_url, self.extra_index_url, self.cache_dir, self.pre
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/pipper.py", line 365, in discover_dependencies_and_versions
req.str(), index_url, extra_index_url, pre, cache_dir
File "/home/andrewv/p3env/pipgrip/lib/python3.6/site-packages/pipgrip/pipper.py", line 292, in _download_wheel
str({package: os.path.join(cache_dir, fname.lstrip(os.path.sep))})
UnboundLocalError: local variable 'fname' referenced before assignment

[ddelange]

with that older pip version, can you try point 3 @andrewv99 ?

[andrewv99]

The modified version you generated is different in more than just the extra diagnostic code it appears, which may account for the differing behaviour. Doing a diff between the version = "0.6.0" and your patched version = "0.6.1.dev1" installations shows other source files have changed too ... can you make a branch of 0.6.0, with only the extra crash logging changes ?

[ddelange]

yes you are right, it contains a tiny leftover from the 0.6.0 release (and the corresponding test).

I just pushed a fix to the branch based on your err.log, would merge and release 0.6.1 if you can confirm it fixes your issue?

[andrewv99]

pip install --force -U git+https://github.com/ddelange/pipgrip.git@verbose-crash
pipgrip --verbose tensorflow==2.0.2
tensorflow==2.0.2
absl-py==0.9.0
six==1.15.0
astor==0.8.1
gast==0.2.2
google-pasta==0.2.0
grpcio==1.31.0
keras-applications==1.0.8
h5py==2.10.0
numpy==1.19.1
keras-preprocessing==1.1.2
opt-einsum==3.3.0
protobuf==3.12.4
setuptools==49.2.1
tensorboard==2.0.2
google-auth-oauthlib==0.4.1
google-auth==1.20.1
cachetools==4.1.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
rsa==4.6
requests-oauthlib==1.3.0
oauthlib==3.1.0
requests==2.24.0
certifi==2020.6.20
chardet==3.0.4
idna==2.10
urllib3==1.25.10
markdown==3.2.2
importlib-metadata==1.7.0
zipp==3.1.0
werkzeug==1.0.1
wheel==0.34.2
tensorflow-estimator==2.0.1
termcolor==1.1.0
wrapt==1.12.1

Many thanks. I will also retest with 0.6.1 when released.

[ddelange]

Thanks for the report!

https://github.com/ddelange/pipgrip/releases/tag/0.6.1

[andrewv99]

Many thanks, confirming ok with 0.6.1:

pip install --force -U pipgrip==0.6.1
pipgrip --verbose tensorflow==2.0.2 2>&1|tee err3.log
tensorflow==2.0.2
absl-py==0.9.0
six==1.15.0
astor==0.8.1
gast==0.2.2
google-pasta==0.2.0
grpcio==1.31.0
keras-applications==1.0.8
h5py==2.10.0
numpy==1.19.1
keras-preprocessing==1.1.2
opt-einsum==3.3.0
protobuf==3.12.4
setuptools==49.2.1
tensorboard==2.0.2
google-auth-oauthlib==0.4.1
google-auth==1.20.1
cachetools==4.1.1
pyasn1-modules==0.2.8
pyasn1==0.4.8
rsa==4.6
requests-oauthlib==1.3.0
oauthlib==3.1.0
requests==2.24.0
certifi==2020.6.20
chardet==3.0.4
idna==2.10
urllib3==1.25.10
markdown==3.2.2
importlib-metadata==1.7.0
zipp==3.1.0
werkzeug==1.0.1
wheel==0.34.2
tensorflow-estimator==2.0.1
termcolor==1.1.0
wrapt==1.12.1