Comments (17)
@mhlavink , I have updated the _importModule
implementation for packages.
This should fix the issue you are observing, and it will have similar behavior as the previous now deprecated module imp.
Try and see if the issue is now fixed.
from yapsy.
I can reproduce this issue.
It seems while migrating away from deprecated Python API's the importing of packages has been broken.
Can you try moving the workflow.yapsy-plugin
within the plugin source directory, to see if it helps?
Something like below layout.
tree:
+ runme
+ plugins/
+ workflow/
+ __init__.py
+ workflow.py
+ workflow.yapsy-plugin
from yapsy.
This workaround works for posted reproducer, but does not work for the app as it has more files in workflow directory.
adding:
from .dwfinfo import DwfInfo
into workflow.py
and creating minimal plugins/workflow/dwfinfo.py with
class DwfInfo:
pass
the result becomes
ERROR:yapsy:Unable to import plugin: /tmp/myapp/plugins/workflow/workflow
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/yapsy/PluginManager.py", line 515, in loadPlugins
candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/yapsy/PluginManager.py", line 587, in _importModule
spec.loader.exec_module(candidate_module)
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/tmp/myapp/plugins/workflow/workflow.py", line 4, in <module>
from .dwfinfo import DwfInfo
ImportError: attempted relative import with no known parent package
from yapsy.
I have attempted to roughly fix the relative import on the latest branch here.
You can try it out quickly using the following steps to see if these changes are resolving the issue you are observing:
# Create a virtual env
python -m venv testvenv
# Enable the virtual env
. ./testvenv/bin/activate
# Install development version of Yapsy with the fix
pip install -e "git+https://github.com/AmeyaVS/yapsy.git@fix_relative_import_in_plugins#egg=yapsy&subdirectory=package"
# Or you can also clone my fork of the yapsy repo checkout the following branch:
git clone https://github.com/AmeyaVS/yapsy -b fix_relative_import_in_plugins
Copy the following sample in a separate directory to see if it is fixing your use-case:
yapsy_test_example.py
from yapsy.PluginManager import PluginManager
def main():
# Load the plugins from the plugin directory.
manager = PluginManager()
manager.setPluginPlaces(["pluginsasdirs2"])
manager.collectPlugins()
# Loop round the plugins and call activate method.
for plugin in manager.getAllPlugins():
plugin.plugin_object.activate()
#plugin.plugin_object.print_name()
if __name__ == "__main__":
main()
Copy the contents of the following directory from the cloned repo:
yapsy/package/test/pluginsasdirs2
Or try this version with your use-case and share your observations.
from yapsy.
Hi, I've tried it and it works with your test and extended reproducer, but it still does not work with the application.
I get two errors, first happens when running reproducer from different directory and can be reproduced with current reproducer - create myapp directory, copy the reproducer into it and then run it:
python3 ./myapp/runme
this results in:
Unable to import plugin: /home/tester/testvenv/myapp/plugins/workflow
Traceback (most recent call last):
File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 515, in loadPlugins
candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 596, in _importModule
parent_module = PluginManager._importModule(parent_name, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 596, in _importModule
parent_module = PluginManager._importModule(parent_name, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tester/testvenv/src/yapsy/package/yapsy/PluginManager.py", line 604, in _importModule
raise ModuleNotFoundError(msg, name=absolute_name)
ModuleNotFoundError: No module named 'myapp'
second kind of error is when I run my app not from source dir, but installed on system:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 515, in loadPlugins
candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 596, in _importModule
parent_module = PluginManager._importModule(parent_name, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 587, in _importModule
absolute_name = importlib.util.resolve_name(plugin_module_name, candidate_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib.util>", line 29, in resolve_name
ImportError: no package specified for '.........usr.lib.python3.12.site-packages.ddashboard.plugins' (required for relative module names)
I was not able to get minimal reproducer for this one yet
from yapsy.
I get two errors, first happens when running reproducer from different directory and can be reproduced with current reproducer - create myapp directory, copy the reproducer into it and then run it: python3 ./myapp/runme
This one I can reproduce, will have to re-work on how the plugins are found and imported.
second kind of error is when I run my app not from source dir, but installed on system:
Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 515, in loadPlugins candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 596, in _importModule parent_module = PluginManager._importModule(parent_name, None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/Yapsy-2.0.0-py3.12.egg/yapsy/PluginManager.py", line 587, in _importModule absolute_name = importlib.util.resolve_name(plugin_module_name, candidate_filepath) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib.util>", line 29, in resolve_name ImportError: no package specified for '.........usr.lib.python3.12.site-packages.ddashboard.plugins' (required for relative module names)
This is seems to be counter intuitive if you already have a package available for a plugin, you could just try importing it directly in your application if the import fails you don't enable the functionality.
My thought w.r.t plugins are somewhat you can drop in the plugins directory a minimal python modules extending the interfaces on your main application. These Python modules are tightly bound to the interfaces on your application.
from yapsy.
@mhlavink , Can you try pulling in the latest changes on the branch?
From what I can observe the first scenario failing for you on running the application from another directory was causing the import failure.
At least what I can observe with my limited collection of plugins and you reproducer steps it seems to be working now.
from yapsy.
sorry for delay, I will test it later today
from yapsy.
I was able to test it again but only the original reproducer works. Extended reproducer (that adds relative import) no longer works and produces error again (same as before):
Traceback (most recent call last):
File "/home/tester2/.local/lib/python3.12/site-packages/yapsy/PluginManager.py", line 516, in loadPlugins
candidate_module = PluginManager._importModule(plugin_module_name, candidate_filepath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tester2/.local/lib/python3.12/site-packages/yapsy/PluginManager.py", line 598, in _importModule
spec.loader.exec_module(candidate_module)
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/home/tester2/reproducer/myapp/plugins/workflow/workflow.py", line 4, in <module>
from .dwfinfo import DwfInfo
ImportError: attempted relative import with no known parent package
this is the updated reproducer I'm using (includes changes from previous comments):
reproducer.tar.gz
from yapsy.
@mhlavink , Can you try and see if you move the workflow.yapsy-plugin
configuration file to it's original location, as per the original application layout?
See if that helps.
from yapsy.
Here's a slightly modified version of the reproducer:
Here's the file layout:
reproducer
├── myapp
│ ├── plugins
│ │ ├── __init__.py
│ │ ├── workflow
│ │ │ ├── dwfinfo.py
│ │ │ ├── __init__.py
│ │ │ └── workflow.py
│ │ └── workflow.yapsy-plugin
│ └── runme
├── runme
└── test.sh
Here's the output captured on the terminal after executing the test.sh
available under the reproducer directory:
DEBUG:yapsy:PluginManagerSingleton initialised
DEBUG:yapsy:PluginFileLocator walks (recursively) into directory: /home/ameya/Downloads/reproducer/myapp/plugins
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:PluginFileLocator found a candidate:
/home/ameya/Downloads/reproducer/myapp/plugins/workflow.yapsy-plugin
DEBUG:yapsy:workflow.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.py is not a valid plugin for strategy info_ext
In DwfInfo
stuff has been imported
In workflow init
DEBUG:yapsy:PluginManagerSingleton initialised
/home/ameya/Downloads/reproducer/./myapp
DEBUG:yapsy:PluginFileLocator walks (recursively) into directory: /home/ameya/Downloads/reproducer/myapp/plugins
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:PluginFileLocator found a candidate:
/home/ameya/Downloads/reproducer/myapp/plugins/workflow.yapsy-plugin
DEBUG:yapsy:workflow.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:workflow.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.cpython-312.pyc is not a valid plugin for strategy info_ext
In DwfInfo
stuff has been imported
In workflow init
the end
DEBUG:yapsy:PluginManagerSingleton initialised
/home/ameya/Downloads/reproducer/myapp/.
DEBUG:yapsy:PluginFileLocator walks (recursively) into directory: /home/ameya/Downloads/reproducer/myapp/plugins
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:PluginFileLocator found a candidate:
/home/ameya/Downloads/reproducer/myapp/plugins/workflow.yapsy-plugin
DEBUG:yapsy:workflow.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.py is not a valid plugin for strategy info_ext
DEBUG:yapsy:__init__.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:workflow.cpython-312.pyc is not a valid plugin for strategy info_ext
DEBUG:yapsy:dwfinfo.cpython-312.pyc is not a valid plugin for strategy info_ext
In DwfInfo
stuff has been imported
In workflow init
the end
Note:
- Get the latest changes from my branch:
https://github.com/AmeyaVS/yapsy/tree/fix_relative_import_in_plugins
from yapsy.
After moving plugin config file to its original location reproducer works. When I try to run the application there is new issue. One plugin uses 3rd party module that runs some slow autodetection on import so it uses lazy import (just import in a method before first use). This import fails now:
from jira import JIRA, resources
with error:
ImportError: cannot import name 'JIRA' from 'jira'
nothing else interesting in traceback. It does not have to be imported in a plugin. Same import just before loading the plugins works, just after that it fails. Adding that import line in current reproducer is not enough to trigger this error
later... ok, found out how to reproduce - there is a jira plugin (Name=Jira Module=jira) and 3rd party module jira that it imports
reproducer3.tar.gz
from yapsy.
So can I consider the issue to be resolved?
Also, naming the plugin as the third-party modules/package name is a little bit concerning.
I am not sure which module, package or namespace are you referring to when in use.
from yapsy.
I think it's still an issue. First, I think there is nothing unusual that plugin that provides for example jira related features is called jira and it uses some 3rd party module. Bigger problem is that module name is locked for everyone.
For example (just tested):
deleted jira plugin
add the import into workflow.py :
from jira import JIRA
create 'requests' plugin
and it fails with:
File "/home/tester/reproducer/myapp/plugins/workflow/__init__.py", line 1, in <module>
from .workflow import Workflow
File "/home/tester/reproducer/myapp/plugins/workflow/workflow.py", line 5, in <module>
from jira import JIRA
File "/home/tester/.local/lib/python3.12/site-packages/jira/__init__.py", line 11, in <module>
from jira.client import (
File "/home/tester/.local/lib/python3.12/site-packages/jira/client.py", line 46, in <module>
from requests import Response
ImportError: cannot import name 'Response' from 'requests' (/home/tester/reproducer/myapp/plugins/requests.py)
as jira import in workflow plugin imports jira.client which tries to import Response from requests, so even the parts that you don't have under control can break this
from yapsy.
@mhlavink , reusing the same module, package names in plugins with 3rd party package would cause issues.
I am not sure how to fix this particular issue you are mentioning.
If you can please share your changes which can help mitigate this issue.
from yapsy.
checked with both reproducer and the app and both work fine 👍
from yapsy.
@mhlavink , thanks for the update.
from yapsy.
Related Issues (8)
- when using except : Exception also log the error message so it is clear why the exception happened
- ConfigurablePluginManager example
- PyPI Release HOT 1
- Maybe it's time to release a new version of Yapsy to PyPi? HOT 1
- Please port to Python3.12 HOT 1
- module uses 'imp' which is deprecated in Python 3.12.x HOT 2
- Request for maintainers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yapsy.