BUG safe_import_context incompatible with typing in CI about benchopt

[francois-rozet]

Currently benchopt install skips the code that is within the context of safe_import_context. This leads to some issues, like the one at hand, but also with imports from benchmark_utils like base classes.

Some features of importlib (like MetaPathFinder) can be used to modify the behavior of import, for instance to create "fake" modules instead of importing them. I think this could be used within safe_import_context to solve these issues.

See

• https://python.plainenglish.io/metapathfinders-or-how-to-change-python-import-behavior-a1cf3b5a13ec
• https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder.

What do you think @tomMoral, @mathurinm, @Badr-MOUFAD ? If you think this is worth it, I can try to write a minimal working example.

[Badr-MOUFAD]

Thanks for flagging this!

I doubt that typed parameters would impair detecting/installing requirements.
Could you pls provide a link to the CI error? Also, a link to the repository would be great!

[francois-rozet]

Hello, https://github.com/JuliaLinhart/benchmark_sbi/actions/runs/5461731758/jobs/9940110771.

Tensor is imported in safe_import_context and used as typing in set_data.

[agramfort]

it's not a typing issue I think but more that pytorch is not properly installed on the SBI benchmark.

[francois-rozet]

It is a typing issue. Switching to the following (outside of the safe_import_context) fixes the CI issue (but it is ugly).

try:
    from torch import Tensor
    from torch.distributions import Distribution
except:
    Tensor = TypeVar("Tensor")
    Distribution = TypeVar("Distribution")

The problem is that benchopt cannot install torch as it cannot even define the class Objective (as Tensor does not exist yet).

[agramfort]

great !

…

Message ID: ***@***.***>

[tomMoral]

This looks indeed promising, creating a fake object automatically when the module is not found would solve many issues.

If you can craft a minimal working example, that would be super helpful

[francois-rozet]

Here is a MWE to replace all imported members (packages, modules, ...) within a context by Mock objects from unittest. Accessing the attributes of a mock object returns a mock object (super cool!), so there is never an issue.

import sys

from contextlib import contextmanager
from importlib.abc import MetaPathFinder, Loader
from importlib.util import spec_from_loader
from unittest.mock import Mock


class MockLoader(Loader):
    def __init__(self, name: str):
        self.name = name

    def create_module(self, spec):
        return Mock(name=self.name)

    def exec_module(self, module):
        module.__path__ = []


class MockFinder(MetaPathFinder):
    def find_spec(self, fullname, path, target=None):
        return spec_from_loader(fullname, MockLoader(fullname))


@contextmanager
def mock_import():
    try:
        sys.meta_path.insert(0, MockFinder())
        yield
    finally:
        sys.meta_path.pop(0)


with mock_import():
    import notapackage  # no error
    from notapackage.notamodule import notamember

import notapackageeither  # error

I think it is only necessary to activate this behavior in benchopt install and when looking for the dataset/solver names and parameters.