Python PDK? about extism

[nilslice]

It's definitely on our radar, and something we would like to make happen. Technically it's already possible to do. The issue is that CPython or another interpreter must be compiled to WASM, and then load python code and run it. So the complexity on both the implementation and well as the plug-in development experience is quite high.

We're still watching the Python community make progress here and are excited for it. If we ship something on the PDK side for Python too soon, it might require users to make really significant changes to their plug-ins.

[nyarly]

I have a Python use case, but not a deep Python knowledge. I understand that Python does sometimes do some compilation to a kind of bytecode? Is there a WASM target for Python instead of a WASM Python host?

[bhelx]

@nyarly

I have a Python use case, but not a deep Python knowledge. I understand that Python does sometimes do some compilation to a kind of bytecode? Is there a WASM target for Python instead of a WASM Python host?

The way this would work would be similar to the JS-PDK: https://github.com/extism/js-pdk
I'm actually debating making that thing universal for all dynamic language like this. But the steps are roughly:

1. Compile the interpreter to Wasm (usually not too hard as most of them are written in C). This singlestore project would do the trick.
2. Load the Wasm program and evaluate the language's standard library source code and and your application source code.
3. Use Wizer to freeze the state of the post-initialized machine into a new Wasm file
4. For each function your plugin exports, compile an equivalent thunk function in Wasm code and expose as a Wasm export. The thunk will essentially grab the VM context and invoke the exported function by name in python/js/whatever.

What comes out of this process is another Wasm file which can be used as an Extism plugin

[z4ce]

I've sometimes wondered if the best path for Python PDK support would be via something like Nuitka or another transpiler. That would avoid having to run the entire python interpreter and let the byte code be far smaller.

[nilslice]

@z4ce - that is an interesting approach! a quick glance through the project docs didn't show much in the form of direct compilation to wasm bytecode though - do you have a link that describes the approach?

If we can get to LLVM IR from Nukita, we could link the extism imports (https://github.com/extism/c-pdk/blob/main/extism-pdk.h#L9-L60), but if there's a custom backend generator, it would take a bit more work. Maybe there's a way to link this into the Python source though and Nukita can handle it?

[z4ce]

Nuitka definitely doesn't have a wasm target right now. I did spend a little bit more time looking at Nuikta and it looks like it relies heavily on libpython -- and I'm not sure how adaptable it might be to being statically linked into a wasm module. It would be interesting to try to compile a program with Nuitka into a statically linked binary with WASI as the target and see what it does.

Another thought could be something like https://github.com/py2many/py2many

It seems to try to convert python to several other languages and does so in more direct way -- but it only supports a subset of python. None of these types of projects seem to be trying to emulate the full standard library or anything.

[atifsyedali]

May we can use https://github.com/RustPython/RustPython to run the plugin?

[bhelx]

I've been doing a few experiments with making a Python PDK. I made some progress using rustpython but want to look at a few more options and commit to getting it done. I've been trying to post in the #python-pdk discord channel when i can if anyone wants to join there and help.

[bhelx]

I pushed up some code using RustPython https://github.com/extism/python-pdk

But I think we still need to strategize on a direction: extism/python-pdk#1

Would love community input or help here.