Implement VCS support about pipgrip

[ddelange]

Released 0.7.0

pipgrip 'freemocap @ git+https://github.com/jonmatthis/freemocap' should now work :)

[ddelange]

Hi @abitrolly,

Indeed, as mentioned in the known caveats section, I haven't had time to properly implement paths and VCS links the same way pip handles them (both in reading and in resolving behaviour).

It is quite tedious, and probably requires a deep dive into pip source (or lots of experimentation) to do properly. I think older versions of pip simply errored once it encountered two different VCS links for the same project name - but not sure if this is still the case with pip's new resolver.

Another implementation hurdle is the fact that PubGrub only supports semver and so will need some edge casing for alpha numeric commit hashes. Same goes for other features that VCS links have in store for us, like egg names. This last one (containing hashtag) will also need some modification of pipgrip's read_requirements function, as it currently splits each line blindly on hashtag to account for inline comments in requirements files.

Just to give an impression of the task, probably more hurdles on the way :)

[abitrolly]

I thought that pipgrip just passes everything to pip and then listens to events.

[ddelange]

pipgrip first builds a list of requested packages, and passes them to pip one at a time recursively to get the informarion to build the dependency tree exhaustively.

To be more precise, this function is the only interaction with pip:
https://github.com/ddelange/pipgrip/blob/0.6.10/src/pipgrip/pipper.py#L380

The rest is all in-house!

[abitrolly]

The linked code calls pip wheel --no-deps ... which can accept URLs. Not https://, but git+https:// works.

$ pip wheel --no-deps git+https://github.com/jonmatthis/freemocap
Collecting git+https://github.com/jonmatthis/freemocap
  Cloning https://github.com/jonmatthis/freemocap to /tmp/pip-req-build-4pk7agii
  Running command git clone -q https://github.com/jonmatthis/freemocap /tmp/pip-req-build-4pk7agii
Building wheels for collected packages: freemocap
  Building wheel for freemocap (setup.py) ... done
  Created wheel for freemocap: filename=freemocap-0.0.17.post0.dev38-py3-none-any.whl size=107546 sha256=304309bf3ebce31fc75d68130af99b6dd9f8f23f355a39d59a89836720b8dcd7
  Stored in directory: /tmp/pip-ephem-wheel-cache-rxy7n0cu/wheels/2a/a7/a4/9608cea0fa42759785d1e30583000552a25f0e898eaba04a89
Successfully built freemocap

So I guess passing the argument as-is should do the trick.

This builds a wheel, which can be inspected by the discover_dependencies_and_versions(). Or where is the problem that I do not see with the wheel above?

[ddelange]

There is no problem with passing a link to pip, the hurdles come beforehand (adding the entry to the pipgrip depencency tree with project name as key) and afterwards (checking for conflict - easier as it needs to be unique).

Beforehand requires effort because discover_dependencies_and_versions is expensive and so gets called lazily, only once pipgrip needs to know details about a specific package version. This means a refactor is needed to rely on that function to provide the project name for VCS projects and local paths, which in turn is needed to add the project to the dependency tree. Normally, project name can be parsed straight from the input, and so the lazy approach works fine.

Afterwards is less tricky: pip install git+https...@master git+https...@a1b2c3 will error even if master happens to point to a1b2c3, so pipgrip should do the same. This means we can check the dependency tree for this project name, if it already exists and we want to add a VCS project to the tree, throw a conflict.

edit: can't throw yet, need to keep different 'versions' in memory so that an incompatibility can be marked (which might trigger backtracking).

Whether this covers all edge cases when VCS projects and/or local paths are encountered, I'm not sure. It would need to be confirmed empirically or by pip source code deep dive, to ensure identical behaviour to pip and avoid non-installable output from pipgrip.

[abitrolly]

Is there no way to feed an URL like https://github.com/jonmatthis/freemocap to pip and get the name and version of the package that it resolves?

[ddelange]

Not without building the wheel (running setup.py) I'm afraid, ref #40 and pypa/pip#1884

But maybe it can be parsed from early pip output and then killed prematurely... 🤔

[ddelange]

But maybe it can be parsed from early pip output and then killed prematurely... 🤔

For pipgrip it doesn't matter though as we'll also need the dependencies for the package sooner or later anyway :)