Numpy example issue about deno_python

[DjDeveloperr]

That is quite weird. I will have to debug this locally so I need some more time, sorry 😅

[hoangpq]

Thanks @DjDeveloperr.

I did something like this in Nodejs https://github.com/hoangpq/pynode but only in prototype stage,
and quite interesting in your project :)

We can use Python ML lib for Deno. Big thank to you and team.

[DjDeveloperr]

So there is only one fix... and it requires a change in Deno itself. We need to pass RTLD_LAZY | RTLD_GLOBAL to dlopen when loading Python shared library so that it's symbols can be imported from subsequently loaded shared libraries, such as NumPy itself in this case.

[DjDeveloperr]

I've come up with a workaround until Deno supports flags in dlopen, that is to open libc and use dlopen from that in order to pass RTLD_GLOBAL flag. I also added a test case for importing NumPy, CI seems passing on the PR

[DjDeveloperr]

Released the fix in 0.1.1

[hoangpq]

Tested on macOS Big Sur, Python 3.8, Deno 1.18.0, deno_python 0.1.1

Thanks for your help 🥇

[DjDeveloperr]

I suspect it is because we do not set the executable path to the actual Python path, and it is likely trying to call deno as python. However this works for me on Windows. I will try to fix ASAP

Edit: indeed it is a macOS/darwin specific issue, I know the fix now

[hoangpq]

Hi @DjDeveloperr.

Thank you for amazing quick response :)

[DjDeveloperr]

Hi!

Can you try if importing deno_python from https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/mod.ts fixes the issue?

[hoangpq]

Hi @DjDeveloperr

It's still happening

[DjDeveloperr]

Hm, can you send the complete error logged this time?

[hoangpq]

Hi @DjDeveloperr.

For sure, here is full stacktrace

Download https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/mod.ts
Download https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/src/python.ts
Download https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/src/ffi.ts
Download https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/src/util.ts
Check file:///Users/hoangpq/WebstormProjects/deno-working/server.ts
error: Uncaught (in promise) PythonError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.8 from "/usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/bin/python3"
  * The NumPy version is: "1.22.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): Symbol not found: _PyBaseObject_Type
  Referenced from: /usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so
  Expected in: flat namespace
 in /usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so

Traceback:
['  File "/usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/__init__.py", line 144, in <module>\n    from . import core\n', '  File "/usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module>\n    raise ImportError(msg)\n']
  throw new PythonError(errorMessage);
        ^
    at maybeThrowError (https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/src/python.ts:643:9)
    at Python.importObject (https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/src/python.ts:731:7)
    at Python.import (https://raw.githubusercontent.com/denosaurs/deno_python/fix-python-exec/src/python.ts:741:17)
    at file:///Users/hoangpq/WebstormProjects/deno-working/server.ts:3:25

[DjDeveloperr]

So the problem here is NumPy is importing symbols using lazy imports (using dlopen with RTLD_NOW) which looks for symbols exported from current process. On unix based systems, this will immediately fail if a symbol is not found in that way. While on Windows, there is nothing like lazy imports and NumPy just opens Python shared library to access those symbols. I will investigate for a possible fix

[hoangpq]

Hi @DjDeveloperr.

Awesome, Thank you for your information 👍

[hoangpq]

Awesome new, @DjDeveloperr 👍