can't load plugins with relative imports anymore about yapsy

[AmeyaVS]

@mhlavink , I have updated the _importModule implementation for packages.
This should fix the issue you are observing, and it will have similar behavior as the previous now deprecated module imp.
Try and see if the issue is now fixed.

[AmeyaVS]

I can reproduce this issue.
It seems while migrating away from deprecated Python API's the importing of packages has been broken.
Can you try moving the workflow.yapsy-plugin within the plugin source directory, to see if it helps?
Something like below layout.

tree:
+ runme
+ plugins/
   + workflow/
       + __init__.py
       + workflow.py
       + workflow.yapsy-plugin

[mhlavink]

This workaround works for posted reproducer, but does not work for the app as it has more files in workflow directory.
adding:
from .dwfinfo import DwfInfo
into workflow.py
and creating minimal plugins/workflow/dwfinfo.py with

class DwfInfo:
    pass

the result becomes

ERROR:yapsy:Unable to import plugin: /tmp/myapp/plugins/workflow/workflow
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/yapsy/PluginManager.py", line 515, in loadPlugins
    candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/yapsy/PluginManager.py", line 587, in _importModule
    spec.loader.exec_module(candidate_module)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/tmp/myapp/plugins/workflow/workflow.py", line 4, in <module>
    from .dwfinfo import DwfInfo
ImportError: attempted relative import with no known parent package

[AmeyaVS]

I have attempted to roughly fix the relative import on the latest branch here.

You can try it out quickly using the following steps to see if these changes are resolving the issue you are observing:

# Create a virtual env
python -m venv testvenv

# Enable the virtual env
. ./testvenv/bin/activate

# Install development version of Yapsy with the fix
pip install -e "git+https://github.com/AmeyaVS/yapsy.git@fix_relative_import_in_plugins#egg=yapsy&subdirectory=package"


# Or you can also clone my fork of the yapsy repo checkout the following branch:
git clone https://github.com/AmeyaVS/yapsy -b fix_relative_import_in_plugins

Copy the following sample in a separate directory to see if it is fixing your use-case:

yapsy_test_example.py

from yapsy.PluginManager import PluginManager

def main():   
    # Load the plugins from the plugin directory.
    manager = PluginManager()
    manager.setPluginPlaces(["pluginsasdirs2"])
    manager.collectPlugins()

    # Loop round the plugins and call activate method.
    for plugin in manager.getAllPlugins():
        plugin.plugin_object.activate()
        #plugin.plugin_object.print_name()

if __name__ == "__main__":
    main()

Copy the contents of the following directory from the cloned repo:
yapsy/package/test/pluginsasdirs2

Or try this version with your use-case and share your observations.

[mhlavink]

Hi, I've tried it and it works with your test and extended reproducer, but it still does not work with the application.

I get two errors, first happens when running reproducer from different directory and can be reproduced with current reproducer - create myapp directory, copy the reproducer into it and then run it:
python3 ./myapp/runme

this results in:

Unable to import plugin: /home/tester/testvenv/myapp/plugins/workflow
Traceback (most recent call last):
  File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 515, in loadPlugins
    candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 596, in _importModule
    parent_module = PluginManager._importModule(parent_name, None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 596, in _importModule
    parent_module = PluginManager._importModule(parent_name, None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 604, in _importModule
    raise ModuleNotFoundError(msg, name=absolute_name)
ModuleNotFoundError: No module named 'myapp'

second kind of error is when I run my app not from source dir, but installed on system:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 515, in loadPlugins
    candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 596, in _importModule
    parent_module = PluginManager._importModule(parent_name, None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 587, in _importModule
    absolute_name = importlib.util.resolve_name(plugin_module_name, candidate_filepath)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib.util>", line 29, in resolve_name
ImportError: no package specified for '.........usr.lib.python3.12.site-packages.ddashboard.plugins' (required for relative module names)

I was not able to get minimal reproducer for this one yet

[AmeyaVS]

I get two errors, first happens when running reproducer from different directory and can be reproduced with current reproducer - create myapp directory, copy the reproducer into it and then run it: python3 ./myapp/runme

This one I can reproduce, will have to re-work on how the plugins are found and imported.

second kind of error is when I run my app not from source dir, but installed on system:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 515, in loadPlugins
    candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 596, in _importModule
    parent_module = PluginManager._importModule(parent_name, None)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 587, in _importModule
    absolute_name = importlib.util.resolve_name(plugin_module_name, candidate_filepath)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib.util>", line 29, in resolve_name
ImportError: no package specified for '.........usr.lib.python3.12.site-packages.ddashboard.plugins' (required for relative module names)

This is seems to be counter intuitive if you already have a package available for a plugin, you could just try importing it directly in your application if the import fails you don't enable the functionality.

My thought w.r.t plugins are somewhat you can drop in the plugins directory a minimal python modules extending the interfaces on your main application. These Python modules are tightly bound to the interfaces on your application.

[AmeyaVS]

@mhlavink , Can you try pulling in the latest changes on the branch?
From what I can observe the first scenario failing for you on running the application from another directory was causing the import failure.
At least what I can observe with my limited collection of plugins and you reproducer steps it seems to be working now.

[mhlavink]

sorry for delay, I will test it later today

[mhlavink]

I was able to test it again but only the original reproducer works. Extended reproducer (that adds relative import) no longer works and produces error again (same as before):

Traceback (most recent call last):
  File "/home/tester2/.local/lib/python3.12/site-packages/yapsy/PluginManager.py", line 516, in loadPlugins
    candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tester2/.local/lib/python3.12/site-packages/yapsy/PluginManager.py", line 598, in _importModule
    spec.loader.exec_module(candidate_module)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/tester2/reproducer/myapp/plugins/workflow/workflow.py", line 4, in <module>
    from .dwfinfo import DwfInfo
ImportError: attempted relative import with no known parent package

this is the updated reproducer I'm using (includes changes from previous comments):
reproducer.tar.gz

[AmeyaVS]

@mhlavink , Can you try and see if you move the workflow.yapsy-plugin configuration file to it's original location, as per the original application layout?

See if that helps.

[AmeyaVS]

Here's a slightly modified version of the reproducer:

reproducer.tar.gz

Here's the file layout:

reproducer
├── myapp
│   ├── plugins
│   │   ├── __init__.py
│   │   ├── workflow
│   │   │   ├── dwfinfo.py
│   │   │   ├── __init__.py
│   │   │   └── workflow.py
│   │   └── workflow.yapsy-plugin
│   └── runme
├── runme
└── test.sh

Here's the output captured on the terminal after executing the test.sh available under the reproducer directory:

DEBUG:yapsy:PluginManagerSingleton initialised
DEBUG:yapsy:PluginFileLocator walks (recursively) into directory: /home/ameya/Downloads/reproducer/myapp/plugins
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:PluginFileLocator found a candidate:
    /home/ameya/Downloads/reproducer/myapp/plugins/workflow.yapsy-plugin
DEBUG:yapsy:workflow.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.py is not a valid plugin for strategy info_ext
In DwfInfo
stuff has been imported
In workflow init
DEBUG:yapsy:PluginManagerSingleton initialised
/home/ameya/Downloads/reproducer/./myapp
DEBUG:yapsy:PluginFileLocator walks (recursively) into directory: /home/ameya/Downloads/reproducer/myapp/plugins
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:PluginFileLocator found a candidate:
    /home/ameya/Downloads/reproducer/myapp/plugins/workflow.yapsy-plugin
DEBUG:yapsy:workflow.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:workflow.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.cpython-312.pyc is not a valid plugin for strategy info_ext
In DwfInfo
stuff has been imported
In workflow init
the end
DEBUG:yapsy:PluginManagerSingleton initialised
/home/ameya/Downloads/reproducer/myapp/.
DEBUG:yapsy:PluginFileLocator walks (recursively) into directory: /home/ameya/Downloads/reproducer/myapp/plugins
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:PluginFileLocator found a candidate:
    /home/ameya/Downloads/reproducer/myapp/plugins/workflow.yapsy-plugin
DEBUG:yapsy:workflow.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:workflow.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.cpython-312.pyc is not a valid plugin for strategy info_ext
In DwfInfo
stuff has been imported
In workflow init
the end

Note:

• Get the latest changes from my branch:
  https://github.com/AmeyaVS/yapsy/tree/fix_relative_import_in_plugins

[mhlavink]

After moving plugin config file to its original location reproducer works. When I try to run the application there is new issue. One plugin uses 3rd party module that runs some slow autodetection on import so it uses lazy import (just import in a method before first use). This import fails now:
from jira import JIRA, resources
with error:

ImportError: cannot import name 'JIRA' from 'jira'

nothing else interesting in traceback. It does not have to be imported in a plugin. Same import just before loading the plugins works, just after that it fails. Adding that import line in current reproducer is not enough to trigger this error

later... ok, found out how to reproduce - there is a jira plugin (Name=Jira Module=jira) and 3rd party module jira that it imports
reproducer3.tar.gz

[AmeyaVS]

So can I consider the issue to be resolved?
Also, naming the plugin as the third-party modules/package name is a little bit concerning.
I am not sure which module, package or namespace are you referring to when in use.

[mhlavink]

I think it's still an issue. First, I think there is nothing unusual that plugin that provides for example jira related features is called jira and it uses some 3rd party module. Bigger problem is that module name is locked for everyone.
For example (just tested):
deleted jira plugin
add the import into workflow.py :
from jira import JIRA

create 'requests' plugin

and it fails with:

 File "/home/tester/reproducer/myapp/plugins/workflow/__init__.py", line 1, in <module>
    from .workflow import Workflow
  File "/home/tester/reproducer/myapp/plugins/workflow/workflow.py", line 5, in <module>
    from jira import JIRA
  File "/home/tester/.local/lib/python3.12/site-packages/jira/__init__.py", line 11, in <module>
    from jira.client import (
  File "/home/tester/.local/lib/python3.12/site-packages/jira/client.py", line 46, in <module>
    from requests import Response
ImportError: cannot import name 'Response' from 'requests' (/home/tester/reproducer/myapp/plugins/requests.py)

as jira import in workflow plugin imports jira.client which tries to import Response from requests, so even the parts that you don't have under control can break this

[AmeyaVS]

@mhlavink , reusing the same module, package names in plugins with 3rd party package would cause issues.
I am not sure how to fix this particular issue you are mentioning.
If you can please share your changes which can help mitigate this issue.

[mhlavink]

checked with both reproducer and the app and both work fine 👍

[AmeyaVS]

@mhlavink , thanks for the update.