Complete port to CPython about importlib_metadata

[jaraco]

In GitLab by @jaraco on Feb 23, 2019, 16:04

@warsaw We're working on a design to help maintain this backport and trying to figure out the best way to:

• keep one canonical source that's applied to CPython and the backport
• rely on Git functionality to maintain ancestry from that canonical source to the CPython repo and backport
• avoid pulling in the entire history into CPython

Here's what I've proposed:

On the left is GitHub (CPython). On the right is this GitLab repo (importlib_metadata). In the middle is a squash of the port, so that commit would appear in the CPython repo and reference (in the commit message) the commit from which it was squashed. From there, that squash commit becomes the canonical source for both the 3.8 implementation and the backport.

One concern about this approach is that CPython doesn't allow for merge commits, only squash merges, meaning that ancestral commit wouldn't exist in the CPython repo after accepting a PR in the default workflow. Would it be possible to bypass the GitHub workflow controls to include that orphan commit so it's available to support the backport?

[jaraco]

In GitLab by @jaraco on Feb 23, 2019, 16:25

assigned to @stevenjma

[jaraco]

In GitLab by @jaraco on Feb 23, 2019, 17:23

@warsaw Do you have any opinion on how the zipp dependency should be handled in the CPython implementation? I see a few possible options:

• vendor it
• supply it as a top-level package under the same name or maybe another name
• merge its functionality into zipfile

[jaraco]

In GitLab by @jaraco on Feb 23, 2019, 20:43

We're going to aim for the least-intrusive "vendor" approach for now.

[jaraco]

In GitLab by @warsaw on Feb 23, 2019, 23:42

We have the same general problem with importlib_resources, but that library is stable enough that it hasn't been much problem to just reapply the diffs from importlib_resources (the canonical version) to CPython.

I expect that eventually importlib_metadata will be as stable, and won't need much work to port changes over to CPython. Maybe that's being too optimistic. The problem I see is that importlib_resources still has to support Python 2.7 (for now - that's part of its mission to obsolete pkg_resources). So somehow, we'll need a repo with 2.7 support and without it. Maybe that's one of the things the middle section will do.

I'm not sure that changing the GitHub workflow for CPython for just this package will be acceptable.

The main thing that I would want in any approach (other that manual, as with importlib_resources) is automation, or well-documented procedures for cross-pollination.

As for the zipp dependency; I'm not a huge fan of vendorizing to be honest. It leads to all kinds of downstream problems. Is zipp something that you would consider adding to Python 3.8, either as a separate library or as part of pathlib? It seems like it would be pretty useful functionality for core Python, and I think you've mentioned that approach before. That would be ideal IMHO.

[jaraco]

In GitLab by @jaraco on Feb 23, 2019, 23:59

@warsaw I think you're right about the code flow in CPython. I'm leaning more toward just maintaining both forms of the code here in this repo and making normal pull requests to CPython by copying from the CPython branch here.

We have another problem - the tests are failing in CPython because neither pip nor importlib_metadata exist as packages. And in the general case, there aren't any third-party packages against which to test.

Do you have any advice on how to handle this based on your past work on the last port? I'm thinking we need to make fixtures that emulate egg-info and dist-info packages, independent of the implicit ones.

[jaraco]

In GitLab by @warsaw on Feb 24, 2019, 24:14

@jaraco That's a good question. I wonder if we can cheat by using the packages vendored in ensurepip? This is probably a good question for python-dev.

[jaraco]

In GitLab by @jaraco on Feb 24, 2019, 22:02

In #49, we've made excellent progress on eliminating dependence on 'pip', 'wheel', and 'importlib_metadata' being installed.

There are still a couple of tests that need to be addressed:

• Some tests use 'packaging' to validate the dependencies. That won't be possible in CPython. I suggest just neutering those tests during the port (see 6c35d56).
• The tests for the 'local' distribution still expect to be running on the local importlib_metadata. We should build a fixture that simulates a package in the current working directory to validate the behavior (see !48).

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 12:58

mentioned in commit c46b705

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 12:58

mentioned in commit 93731b5

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 12:58

mentioned in commit 2c734fe

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 12:58

mentioned in commit 5b37e57

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 12:58

mentioned in commit 7926e6f

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 12:58

mentioned in commit 44a29b7

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 13:17

In da98d7d, I've brought over the work by @stevenjma and merged it with the work in master.

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 13:20

I wonder if we can cheat by using the packages vendored in ensurepip?

I'm pretty sure we don't actually want pip. I only chose to use it because it was an installed 'dist-info' package. The work that @stevenjma has done in the cpython branch to create fixtures makes this project (and the forward-port) able to run tests without pip at all.

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 13:29

assigned to @jaraco and unassigned @stevenjma

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 13:46

Now that we have more code in place, I've identified a few more issues:

• The stub I created for packaging.requirements isn't sufficient to fully address the tests' expectations.
• At least one test depends on importlib.metadata.__version__, which isn't meaningful in the context of the stdlib.
• The test named test_api conflicts with the existing test in the stdlib (and tests aren't namespaced).

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 13:47

Also, many tests in test_util start to fail with:

ERROR: test_find_relative_module (test.test_importlib.test_util.Frozen_FindSpecTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jaraco/code/public/cpython/Lib/test/test_importlib/test_util.py", line 505, in test_find_relative_module
    with util.temp_module(name, pkg=True) as pkg_dir:
  File "/Users/jaraco/code/public/cpython/Lib/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Users/jaraco/code/public/cpython/Lib/test/test_importlib/util.py", line 153, in temp_module
    invalidate_caches()
  File "/Users/jaraco/code/public/cpython/Lib/importlib/__init__.py", line 71, in invalidate_caches
    finder.invalidate_caches()
TypeError: invalidate_caches() missing 1 required positional argument: 'self'

• finder.invalidate_caches missing argument (fixed in 8531ba6).

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 13:50

Here's how I'm running the tests:

• Check out importlib_metadata to the cpython branch (if using an existing clone, run git clean -fdx to clean it).
• Check out cpython to the master branch (soon this will be '3.8').
• Build cpython
• Copy the files from importlib_metadata to cpython; i.e. cp -r ~/p/importlib_metadata/* .
• Run the tests with ./python.exe -m tests.importlib -v.

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 14:50

mentioned in commit e239c9b

[jaraco]

In GitLab by @jaraco on Mar 25, 2019, 15:00

I think the errors about "invalidate_caches" are pointing to a design flaw in the importlib_metadata implementation. Probably importlib_metadata should be creating instances of MetaPathFinders on sys.meta_path, rather than adding classes with class methods.

[jaraco]

In GitLab by @jaraco on Mar 26, 2019, 02:06

mentioned in merge request !50

[jaraco]

In GitLab by @jaraco on Mar 26, 2019, 02:32

The port is largely complete and submitted as cpython 12547. More revisions may be necessary, but the work is largely complete.

[jaraco]

In GitLab by @jaraco on Mar 26, 2019, 02:32

closed

[jaraco]

In GitLab by @jezdez on Apr 18, 2019, 17:03

Huzzah! 😍