OSError [WinError 126] [2496] about pyinstaller-hooks-contrib

[hyunjung0208]

This is output

4MMDETECTION.txt

[eric15342335]

hmm.. the torch hook should be able to do it

[eric15342335]

What is your build log?

[eric15342335]

@hyunjung0208 You can add --collect-binaries torch to PyInstaller as an temporary solution.

---

I believe the get_package_paths() and add it to datas is not a good method, as it will miss dynamic libraries's dependencies.
As shown in the warning:

WARNING: lib not found: torch_python.dll dependency of E:\development\Anaconda\envs\pyi-testenv\lib\site-packages\torch\_C.cp39-win_amd64.pyd

pyinstaller-hooks-contrib/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-torch.py

Line 15 in 86a6eb7

datas = [(get_package_paths('torch')[1],"torch"),]

[bwoodsend]

Where is that torch_python.dll file then if it's not in conda/bin or in site-packages/torch?

[hyunjung0208]

@eric15342335
I did your recommend, and try pyinstaller 4mmdetection.spec
but it can't install and also has error.
"NameError: name 'get_package_paths' is not defined"
What else can I do to solve this problem?
Just in caseI added from PyInstaller.utils.hooks import get_package_paths to the py file, and then tried a new pyInstaller.

@hyunjung0208 You can add --collect-binaries torch to PyInstaller as an temporary solution.

I believe the get_package_paths() and add it to datas is not a good method, as it will miss dynamic libraries's dependencies.
As shown in the warning:

WARNING: lib not found: torch_python.dll dependency of E:\development\Anaconda\envs\pyi-testenv\lib\site-packages\torch\_C.cp39-win_amd64.pyd

pyinstaller-hooks-contrib/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-torch.py

Line 15 in 86a6eb7

datas = [(get_package_paths('torch')[1],"torch"),]

[rokm]

@hyunjung0208
Can you try manually copying c:\users\kim\anaconda3\envs\mm\Library\bin\uv.dll to your program's dist directory, i.e., dist\4mmdetection?

[hyunjung0208]

Where is that torch_python.dll file then if it's not in conda/bin or in site-packages/torch?

@bwoodsend
after pyinstaller, it is in dist/4mmdetection/torch/lib/
And

in C:\Users\kim\anaconda3\envs\mm\Lib\site-packages\torch\lib
in C:\Users\kim\anaconda3\pkgs\pytorch-1.9.0-py3.7_cuda10.2_cudnn7_0\Lib\site-packages\torch\lib\

Do I need to add torch-related path to the environmental variable?

[hyunjung0208]

@hyunjung0208
Can you try manually copying c:\users\kim\anaconda3\envs\mm\Library\bin\uv.dll to your program's dist directory, i.e., dist\4mmdetection?

@rokm @
I manually copied that, and run -> ModuleNotFoundError: No module named 'mmcv._ext' [5096]Failed to execute script '4mmdetection' due to unhandled exception!
So
I revised 4mmdetection.spec hiddenimports=['mmcv._ext']
and run -> IT has same error
[ERROR]
OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\username\scar2\dist\4mmdetection(file name)\torch\lib\torch_python.dl l" or one of its dependecies.
[11176] Failed to execute script '4mmdetection' due to unhandled exception!

[rokm]

Have you manually copied the uv.dll again after revising the 4mmdetection.spec and rebuilding?

[eric15342335]

Just in caseI added from PyInstaller.utils.hooks import get_package_paths to the py file, and then tried a new pyInstaller.

No you don't need to. I'm talking about the hook.
Hmmm, did pyinstaller --collect-binaries torch 4mmdetection.py work? It works for me with an import torch

[hyunjung0208]

Have you manually copied the uv.dll again after revising the 4mmdetection.spec and rebuilding?

Oh, I made a mistake there.
Copy uv.dll manually again, so it runs well. Thank you so much.
By the way, may I know how you found this error?
May I know what uv.dll is?
When I try to run this program on another computer, I'm asking this question because I think this uv.dll is causing problems.
Is there a way to copy the program file with usb and run it on another computer?

[rokm]

By the way, may I know how you found this error?

OSError: [WinError 126] The specified module could not be found. Error loading "XYZ" or one of its dependecies. usually means that there's a DLL missing. It usually helps to inspect the python extensions and other DLLs in using Dependency Walker, and see what's missing.

May I know what uv.dll is?

It's a shared library that comes from the libuv package, one of packages that pytorch depends on.

When I try to run this program on another computer, I'm asking this question because I think this uv.dll is causing problems.
Is there a way to copy the program file with usb and run it on another computer?

If the frozen program does not work on another computer, then you could be missing some other DLLs that were present on your build computer (or in its conda environment, even).

Like @eric15342335 said, the problem is that our pytorch hook collects everything as data files, so shared libraries do not get properly analyzed for their dependencies. With pytorch installed from pip, this is not so much of a problem, because all dependencies are part of the wheel. But with anaconda installation, some of the dependencies are shared and installed outside of the package's location, and due to lack of dependency analysis, we end up missing them (this is what happens with the uv.dll and that's why you need to copy it manually).

[hyunjung0208]

Like @eric15342335 said, the problem is that our pytorch hook collects everything as data files, so shared libraries do not get properly analyzed for their dependencies. With pytorch installed from pip, this is not so much of a problem, because all dependencies are part of the wheel. But with anaconda installation, some of the dependencies are shared and installed outside of the package's location, and due to lack of dependency analysis, we end up missing them (this is what happens with the uv.dll and that's why you need to copy it manually).

@rokm

I really appreciate your detailed explanation.
So if I install the pythorch as a pip instead of conda, I don't need to manually copy uv.dll?
When I did pip install torch, there was an error, so I did pip install torch.
Then, the torch version comes out as 1.9.0+cpu, so you can't use cuda and gpu, but I'm going to ignore it for now and try to proceed.
I installed torchvision as conda, but if I don't want to manually copy uv.dll, do I have to install it as a pip?

[rokm]

I think it should work with torch installed via pip. But I would generally recommend against mixing conda and pip. I.e., stick to a pure conda environment, or install python from python.org, and use venv and pip.

That said, if you don't want to manually copy the uv.dll in your conda-based installation, you could copy it from
c:\users\kim\anaconda3\envs\mm\Library\bin\uv.dll
into
c:\Users\kim\anaconda3\envs\mm\Lib\site-packages\torch\lib
and it should be collected automatically, along with other files from that directory.

Or you could collect the missing libraries as part of extra pre-processing in your .spec file (the example below is for CUDA10-enabled version):

# PyInstaller spec file

import ctypes.util
extra_torch_dlls = [
    # UV
    'UV.DLL',
    # CUDA
    'CUSPARSE64_10.DLL',
    'CURAND64_10.DLL', 
    'CUSOLVER64_10.DLL',
    'NVTOOLSEXT64_1.DLL',
    'CUFFT64_10.DLL',
    'CUBLAS64_10.DLL',
    'CUBLASLT64_10.DLL',
    'NVRTC64_102_0.DLL',
]

binaries = []
for dll in extra_torch_dlls:    
    dll_file = ctypes.util.find_library(dll)
    if dll_file:
        binaries += [(dll_file, '.')]

a = Analysis(...
             binaries=binaries,  # pass the extra collected binaries to PyInstaller's Analysis
             ...)

# the rest of the spec file...