mnmelo / lazy_import Goto Github PK

Ordinarily, accessing attributes on a LazyModule triggers the import, but sometimes this can happen unexpectedly. One such case happens with the __file__ attribute. Here is a minimal example.

fail.py

raise ValueError('Boom!')

script.py

import inspect
from lazy_import import lazy_module

fail = lazy_module('fail')
frame = inspect.currentframe()
mod = inspect.getmodule(frame)

Running the above script triggers the import (and hence, the error). This results from inspect checking the __file__ attribute of every module in sys.modules, including fail. The behavior is undesirable since the user did not intentionally access any attribute of fail.

To fix this, one could simply add '__file__' to the list of exemptions in LazyModule.__getattribute__.

This might be of interest as a limitation that you might considder documenting.

Trying to use autoreload in a Jupyter notebook then lazy-import does not work...

%load_ext autoreload
%autoreload 2

...

Error in callback <bound method AutoreloadMagics.post_execute_hook of <autoreload.AutoreloadMagics object at 0x0000000004CE1908>> (for post_execute):

...

ImportError: ... attempted to use a functionality that requires module ..., but it couldn't be loaded. Please install features and retry.

lazy_module breaks built-in modules (i.e. modules listed in sys.builtin_module_names) that haven't been loaded yet:

>>> import lazy_import
>>> math = lazy_import.lazy_module('math')
>>> math.log
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ring/.local/lib/python3.6/site-packages/lazy_import/__init__.py", line 157, in __getattribute__
    return super(LazyModule, self).__getattribute__(attr)
AttributeError: module 'math' has no attribute 'log'

The same happens, for example, with zlib and pwd (but not with sys, because it's already loaded).

lazy_module changes root logger and thus affects programs importing it in undesired way.
The following snippet illustrates the issue:

import lazy_import
import logging

if __name__ == '__main__':
    logger = logging.getLogger('dupa')

    logging.basicConfig(
        format='%(asctime)s ai [%(process)d] <%(levelname)s> '
               '| %(module)s | %(processName)s | %(message)s',
        datefmt='%Y-%m-%d %H:%M:%S',
        level=logging.DEBUG
    )

    logger.info('DUPA')

Removing the import lazy_import restores desired default behavior.

lazy_import should use its own logger and refrain from calling logging.basicConfig().

As per https://docs.python.org/3/library/logging.html#logging.basicConfig

This function does nothing if the root logger already has handlers configured for it.

Code I used: (reproduce)

import lazy_import
ujson = lazy_import.lazy_module("ujson")
print(dir(ujson))
# Returns ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
ujson.dump()
# AttributeError

Ujson when lazy imported is basically blank. This works with many other libraries, just not ujson.
Lazy_import version 0.2.2, Windows 10, Python 3.11.4

Even if I place try/except around the lazy statements or the lazy_callable routine -- I can't catch the errors and I get: the standard: ImportError: XXX attempted to use a functionality that requires model aaa, but it couldn't be loaded.

I'd like to have my main routine still run if the lazy loading fails.

Maybe it's because my main routine is within a Flask app. Hmm, maybe I should post a stacktrace..

If I mark a sub module of a sub module as lazy like so:

>>> import lazy_import

>>> lazy_import.lazy_module("wx.lib.masked")
Lazily-loaded module wx.lib.masked

>>> import wx.lib.masked

It works fine. For example:

>>> print(wx)
Lazily-loaded module wx

>>> print(wx.lib.masked)
Lazily-loaded module wx.lib.masked

But, if I load a different sub module of that same sub module like so:

>>> import wx.lib.buttons

Then, the sub sub module marked as lazy loaded is not available. For example:

>>> print(wx.lib.buttons)
<module 'wx.lib.buttons' from '...\\lib\\site-packages\\wx\\lib\\buttons.py'>

>>> print(wx.lib.masked)
Traceback (most recent call last):
  File "<pyshell#11>", line 1, in <module>
    print(wx.lib.masked)
AttributeError: module 'wx.lib' has no attribute 'masked'

If I do not do the lazy loading, then it works fine. For example,

>>> import wx.lib.masked

>>> print(wx)
<module 'wx' from 'M:\\Versions\\Python_DMTE_dev\\lib\\site-packages\\wx\\__init__.py'>

>>> print(wx.lib.masked)
<module 'wx.lib.masked' from 'M:\\Versions\\Python_DMTE_dev\\lib\\site-packages\\wx\\lib\\masked\\__init__.py'>

>>> import wx.lib.buttons
>>> print(wx.lib.buttons)
<module 'wx.lib.buttons' from 'M:\\Versions\\Python_DMTE_dev\\lib\\site-packages\\wx\\lib\\buttons.py'>

>>> print(wx.lib.masked)
<module 'wx.lib.masked' from 'M:\\Versions\\Python_DMTE_dev\\lib\\site-packages\\wx\\lib\\masked\\__init__.py'>

Loving your lazy_import package. Thanks for the work!

TL;DR could we have a scenario where package imports Class1 lazily but allows other direct imports of that Class1 not to be lazy? Maybe it's possible but I couldn't see how. Generally I like that the lazy_import has its effects globally, but at times I'd like that not to be the case.

The issue I'm having, that's currently limiting my use of it, is the fact that it prevents any use of subclassing, as far as I can tell. I understand why this is the case (the fact that we import a wrapper for the object rather than the actual object) but as far as I can see there are no workarounds for people that really need the subclass.

Consider the structure (in a large project):
package.subpackage1.Class1
package.subpackage2.Class2

I would like to allow the user two routes to access the classes:

• from package import Class1 (so within package I do lazy_import.lazy_class("subpackage1.Class1"))
• but I'd also like to allow from package.subpackage1 import Class1 for a true import of the class allowing subclassing

Why?

• Class1 and Class2 are the main classes in their subpackages (separated into different files to keep files manageable) so it's convenient to make them directly available from main_package.
• They are also large so we don't always want to import them if the user won't use them
• Occasional users do want to subclass though. If there's no way to do this after setting the lazy_import code then it's a deal-breaker

This lazy import looks like the best match for our needs but returns an error when installing. Running on python 3.6.4 or 3.5.2.

pip install lazy_module
Collecting lazy_module
Could not find a version that satisfies the requirement lazy_module (from versions: )
No matching distribution found for lazy_module

In my _init_.py , I have two subpackages that I'm trying to lazy load.

lazy1 = lazy_import.lazy_module("mypkg.sub1")
lazy2 = lazy_import.lazy_module("mypkg.sub2")

In an IDE when I go to load something from lazy1, it also loads lazy2. I would like them to lazily load separately. In each subpackage, I have an _init_.py where I do some relative imports. I'm seeing very odd behavior / crashing in this setup that I think would be fixed if the lazy imports would load separately. ``

ImportError: mmskeleton.utils.third_party attempted to use a functionality that requires module mmdet.apis, but it couldn't be loaded. Please install mmdet and retry.

Two issues I am seeing in Python 2.7:

1. Only the first lazy submodule will appear in a package shared by two lazy submodules.

la = lazy_import.lazy_module('package.a')
lb = lazy_import.lazy_module('package.b')

import package
'a' in type(package)._lazy_import_submodules
'b' not in type(package)._lazy_import_submodules

This is because when a cached lazy package is found (https://github.com/mnmelo/lazy_import/blob/master/lazy_import/__init__.py#L378), we don't update the reference of _LazyModule to the cached module and hence the module from the last iteration of the loop has it's _lazy_import_submodules attribute updated. Thus resulting in the following:

'b' in type(lb)._lazy_import_submodules

2. After a package is loaded, references to lazy modules are lost, resulting in:

la = lazy_import.lazy_module('package.a')

import package
package.a == la
from package import a  !!!! fails !!!!

This problem is rooted here: https://github.com/mnmelo/lazy_import/blob/master/lazy_import/__init__.py#L689

The issue is that prior to reloading the module, we clean all lazy sub-module attributes and LazyModule class attributes. We then try to restore those attributes, but we rely on a class attribute which has been removed. Thus no sub-modules are found and the lazy state isn't restored.

To expose this error and compare with options that work:
Download the two files
Change the extensions back to .py
Save child.py in a folder Cmd and parent.py in subfolder Cmd.ribs. The Cmd name is not important for the error.
Run child.py from shell prompt. (I'm using Mac OS X Sierra.)
The part that exposes the error is ready. Rem out and in the three line tests in parent and the associated parts in child to expose options that work with lazy_module and system importing and with pre-importing cmd before calling lazy_callable.

This error was occurring before error #2 was fixed. This error is somewhat opposite error #2. In this case, adding "import cmd" before lazy_callable('cmd.Cmd') avoids the error.

I thought the error might have something to do with my choice of project root Cmd but it occurs with other root folder names.

I hope this helps.
child.txt
parent.txt

This first occurred in my modules but the error also occurs with python email module. The error occurs when email is system imported before lazy_module('email.mime') or lazy_module('email.mime.audio') or email.mime is imported before lazy_module('email.mime.audio').

Ceti$ py
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

from lazy_import import lazy_module
import sys
'email' in sys.modules
False
lazy_module('email.mime.audio') # This works
^D

Ceti$ py
Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

from lazy_import import lazy_module
import email
import sys
'email' in sys.modules
True
lazy_module('email.mime.audio') # This fails first time, works on second call
Traceback (most recent call last):
File "", line 1, in
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/lazy_import/init.py", line 302, in lazy_module
mod = _lazy_module(modname, error_strings, lazy_mod_class)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/lazy_import/init.py", line 354, in _lazy_module
super(LazyModule, mod).setattr(submodname, submod)
TypeError: super(type, obj): obj must be an instance or subtype of type

Are you planning to put lazy_import on conda-forge? I would like to use it in my operational library, but conda-forge does not handle pip install with setuptools.

Hello

Easy import works great and does a marveillous job, however I had to fork and disable the VERSION file trick in order to package it properly in pyinstaller, since this guy does not take full package but extract only python files from the dependencies.

Can you update to be pyinstaller friendly? Lazy import is really cool to avoid unneeded module load in CLI tools using pyinstaller.

Thanks!

Hi,

I think it would be great if help(lazy_import.lazy_callable('pkg.mod', 'func')) returned a docstring along the lines of 'lazy callable object pkg.mod.func', currently I get the less helpful 'Class for lazily-loaded callables that triggers module loading on access\n\n '

Collecting lazy_import
  Downloading https://files.pythonhosted.org/packages/44/2e/5378f9b9cbc893826c2ecb022646c97ece9efbaad351adf89425fff33990/lazy_import-0.2.2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-084ivfpp/lazy-import/setup.py", line 6, in <module>
        readme = infile.read()
      File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 621: ordinal not in range(128)

On some systems pip install lazy_import fails with the above error. I guess they're not defaulting to UTF-8 decoding.

Please include the license file in your PyPI distribution to make it redistributable.