Coder Social home page Coder Social logo

componentize-py's Introduction

componentize-py

A Bytecode Alliance project

This is a tool to convert a Python application to a WebAssembly component. It takes the following as input:

  • a WIT file or directory
  • the name of a WIT world defined in the above file or directory
  • the name of a Python module which targets said world
  • a list of directories in which to find the Python module and its dependencies

The output is a component which may be run using e.g. wasmtime.

Getting Started

First, install Python 3.10 or later and pip if you don't already have them. Then, install componentize-py:

pip install componentize-py

Next, create or download the WIT world you'd like to target, e.g.:

cat >hello.wit <<EOF
package example:hello;
world hello {
  export hello: func() -> string;
}
EOF

If you're using an IDE or just want to examine the bindings produced for the WIT world, you can generate them using the bindings subcommand:

componentize-py -d hello.wit -w hello bindings hello_guest

Then, use the hello module produced by the command above to write your app:

cat >app.py <<EOF
import hello
class Hello(hello.Hello):
    def hello(self) -> str:
        return "Hello, World!"
EOF

And finally generate the component:

componentize-py -d hello.wit -w hello componentize --stub-wasi app -o app.wasm

To test it, you can install wasmtime-py and use it to generate host-side bindings for the component:

pip install wasmtime
python3 -m wasmtime.bindgen app.wasm --out-dir hello_host

Now we can write a simple host app using those bindings:

cat >host.py <<EOF
from hello_host import Root
from wasmtime import Config, Engine, Store

config = Config()
config.cache = True
engine = Engine(config)
store = Store(engine)
hello = Root(store)
print(f"component says: {hello.hello(store)}")
EOF

And finally run it:

 $ python3 host.py
component says: Hello, World!

See the examples directories for more examples, including various ways to run the components you've created.

Known Limitations

Currently, the application can only import dependencies during build time, which means any imports used at runtime must be resolved at the top level of the application module. For example, if x is a module with a submodule named y the following may not work:

import x

class Hello(hello.Hello):
    def hello(self) -> str:
        return x.y.foo()

That's because importing x does not necessarily resolve y. This can be addressed by modifying the code to import y at the top level of the file:

from x import y

class Hello(hello.Hello):
    def hello(self) -> str:
        return y.foo()

This limitation is being tracked as issue #23.

See the issue tracker for other known issues.

Contributing

See CONTRIBUTING.md for details on how to contribute to the project and build it from source.

componentize-py's People

Contributors

dicej avatar karthik2804 avatar kyleconroy avatar peterhuene avatar sneakyberry avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

componentize-py's Issues

Type checking fails when records are defined in a world with functions

I think this is actually a bug but it may not be one that idiomatic wit files would trigger so it may not be an important issue but I'll post it anyway just in case.

When records are defined in a world along side functions which use the record type, the functions are generated first in the binding's init.py file so that type checking fails due to the undefined type.

For example, this wit file will not produce usable bindings:

package example:broken;

world broken-world {
    record a-record {
        field: s32
    }

    import get-a-record: func() -> a-record;
    export do-stuff: func() -> a-record;
}

My naive fix was to stick from __future__ import annotations at the top of the __init__.py file and that does work. But then I realized that the idiomatic thing to do seems to be to put type definitions in a separate types interface and import from there. Doing that produces valid bindings without having to change componentize_py at all.

Running example

I'm looking to get the example in the readme working. I have the component wasm compiled via this command:

$ componentize-py -d hello.wit -w hello componentize app -o app.wasm

Now I want to run this via wasmtime but that gives me this error:

$ wasmtime run --wasm component-model app.wasm                             
Error: failed to run main module `app.wasm`

Caused by:
    exported instance `wasi:cli/[email protected]` not present

I thought I could convert this to a core module with wasm-tools but that gives me a different error:

$ wasm-tools component new -v app.wasm -o out.wasm --adapt wasi_snapshot_preview1.reactor.wasm
error: failed to encode a component from module

Caused by:
    0: unknown table 2: exported table index out of bounds (at offset 0xb)

Ideally I'd like to take Python and covert it into wasm much like how it's done in Javy (https://github.com/bytecodealliance/javy) where the output of javy compile is code that I can execute with wasmtime <output>.wasm.

Thanks for any help or ideas! =)

Add support for type aliases

Currently, componentize-py generates no code for type aliases; instead, it just uses the dealiased types directly. Instead, we should generate assignment expressions for each alias and use the alias in all the same places (e.g. function parameter types, etc.) that the original WIT files used them.

List errors which might be raised in function docstrings

Python does not currently have a way to represent the types of errors/exceptions which might be raised by a function in a typed function signature, so the next best thing is to list them in the docstring (e.g. following any docs derived from WIT doc comments).

Generated component has dependency on wasi:cli/[email protected]

I'm attempting to port the CompontenizeJS example to compotentize-py.

package local:hello;

world hello {
  export hello: func(name: string) -> string;
}
componentize-py -d hello.wit -w hello bindings .
import hello

class Hello(hello.Hello):
    def hello(self, name) -> str:
        return f"Hello, {name}, from Python!"

Trying to run the component fails

componentize-py -d hello.wit -w hello componentize app -o hello.component.wasm
cargo build --release
./target/release/wasmtime-test

Error: import `wasi:cli/[email protected]` has the wrong type

Caused by:
    0: instance export `get-environment` has the wrong type
    1: expected func found nothing
make: *** [py-hello] Error 1

re-init random seed(s) on resume

We're currently using component-init to snapshot the state of an initialized app, and the initialization process may involve getting random numbers from the host and using them to seed one or more PRNGs. That could pose a security risk if we don't force those PRNGs to be reseeded on resume, so we should find these cases in wasi-libc and/or CPython. This could also be an issue in Python code and/or native extensions, but it's not clear how we would address that; we may need to leave that up to the application programmer.

Add contributor documentation

This should include a CONTRIBUTING.md file, plus an ARCHITECTURE.md describing the high-level architecture of the project. In addition, each source file needs doc and implementation comments.

pyo3_runtime.PanicException: internal error: entered unreachable code

I ran into the following issue:

thread '<unnamed>' panicked at 'internal error: entered unreachable code', src/lib.rs:450:35
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/eliabieri/Library/Caches/pypoetry/virtualenvs/widget-4qY9vehV-py3.11/bin/componentize-py", line 8, in <module>
    sys.exit(script())
             ^^^^^^^^
pyo3_runtime.PanicException: internal error: entered unreachable code

Steps to reproduce

Prerequisites: Poetry

  1. Clone repo
    git clone --recurse-submodules [email protected]:eliabieri/wg_display_widget_py.git
  2. Install dependencies
    poetry install
  3. Generate bindings
    poetry run componentize-py --wit-path wg_display_widget_wit/wit --world widget bindings .
  4. Generate component
    poetry run componentize-py --wit-path wg_display_widget_wit/wit --world widget componentize --output widget.wasm --python-path widget widget

Broken http example: import error from poll_loop.py: StreamErrorClosed

Attempting an example for the first time, and it looks like there's a typo in the example code.

I run componentize-py -d ../../wit -w wasi:http/[email protected] componentize app -o http.wasm and get this traceback:

Traceback (most recent call last):
  File "/home/daedalus/src/wasm-experiments/.venv/bin/componentize-py", line 8, in <module>
    sys.exit(script())
AssertionError: Traceback (most recent call last):
  File "/0/app.py", line 10, in <module>
    import poll_loop
  File "/bundled/poll_loop.py", line 17, in <module>
    from proxy.imports.streams import StreamErrorClosed, InputStream
ImportError: cannot import name 'StreamErrorClosed' from 'proxy.imports.streams' (/world/proxy/imports/streams.py). Did you mean: 'StreamError_Closed'?

It looks like bundled/poll_loop.py does indeed try to import and use StreamErrorClosed instead of StreamError_Closed, but when I try to make the change locally (so I can maybe file a PR), it doesn't appear to fix the error.

I clearly don't understand how Python imports work with wasmtime and/or componentize-py. What am I missing? Is it pulling in the import from another path or a prebuilt module somewhere?

No rush, just wanted to start learning about wasm and coming from my background with Python seemed a reasonable entry point.

Running component without WASI

Context

I have a Rust project that loads and executes wasm components using wasmtime.
The wasmtime usage can be seen here.

If I try to instantiate a component generated using componentize-py, I get an error concerning a type mismatch with a WASI type

import `wasi:clocks/wall-clock` has the wrong type

Questions

  • Am I forced to use WASI when I want to run a wasm component generated by componentiue-py?
    • If yes, how would I configure wasmtime?

Apologies if these questions should be clear from the documentation. I was just not able to figure these out, even after trying for some time.

Links

Unresolved symbols when building from source

I'm trying to build the repo from source following the instructions in CONTRIBUTING.md and it works, including running cargo run -- --help and also the bindings command but when I try to run componentize I get the following error:

> cargo run --release -- -d hello.wit -w hello componentize app -o app.wasm
warning: [email protected]: using seed f0dd8d4b7597bc894379cc4d111aed752baa62cfc70fb22e8d4bb2810a9ade4b (set COMPONENTIZE_PY_TEST_SEED env var to override)
warning: [email protected]: using count 10 (set COMPONENTIZE_PY_TEST_COUNT env var to override)
warning: field `package` is never read
  --> src/summary.rs:76:9
   |
74 | pub struct MyInterface<'a> {
   |            ----------- field in this struct
75 |     pub id: InterfaceId,
76 |     pub package: Option<PackageName<'a>>,
   |         ^^^^^^^
   |
   = note: `MyInterface` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis
   = note: `#[warn(dead_code)]` on by default

warning: `componentize-py` (lib) generated 1 warning
    Finished `release` profile [optimized] target(s) in 0.26s
     Running `target/release/componentize-py -d hello.wit -w hello componentize app -o app.wasm`
Error: unresolved symbol(s):
	libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
	libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
	libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)

When I build the package with maturin and install it I see the same error but the version installed using pip works in the same virtualenv.

I'm on Arch linux with Python 3.11.8 and the latest nightly for rust stuff.

Improve type mismatch detection and reporting

Currently, if Python code passes the wrong number or types of parameters to the host, or returns a result of an unexpected type, componentize-py generally reacts by panicking and trapping. That's probably unavoidable in most cases, but it should at least print an informative and actionable diagnostic prior to panicking.

Download dependencies automatically

Per yesterday's Python guest tooling meeting, we'd like componentize-py to download any dependencies found in the application's pyproject.toml (including transitive dependencies), placing them in a temporary directory for use while building the component.

Similar to micropip, componentize-py should grab any pure Python packages from PyPI, only looking to an alternative repo for packages with native extensions. We (i.e. the Bytecode Alliance) will need to maintain this repo and populate it with WASI builds of popular projects until WASI wheels are standardized and published upstream to PyPI.

To start with, componentize-py could shell out to e.g. pip --install --platform wasi-sdk-22 --abi cp311 --root <tmpdir>, but eventually we'd like to make it fully self-contained (i.e. not require Python or pip to be installed).

Run MyPy on all test code and bindings

In addition to running the test suite, we should dump the app code and generated bindings in a directory and run MyPy on them, asserting that no errors are found. This will help us catch any regressions in the type annotations,

time.sleep sleeps indefinitely

For example, when targeting the wasi:cli command world:

from hello import exports
from time import sleep

class Run(exports.Run):
    def run(self):
        print("Hello, world!")
        sleep(0.00001)

The above prints "Hello, world!" and then seems to sleep forever.

Allow dynamic imports in sandbox example

I'd like to execute code using the sandbox example, but am running into an issue. I want to allow exec'd code to import packages, but if I don't import packages in the host script, the guest code won't run.

# Guest code
import json

print(json.dumps({"message": "Hello"}))
# Works
import sys
import json

from command import exports

class Run(exports.Run):
    def run(self):
        with open(sys.argv[1]) as f:
            code = f.read()
            exec(code)
# Fails
import sys

from command import exports

class Run(exports.Run):
    def run(self):
        with open(sys.argv[1]) as f:
            code = f.read()
            exec(code)

Remove init function and types from the final output

The pre-initialization process currently involves an exported anonymous interface and types (defined in init.wit). That should be stripped out of the final component since it's not part of the application's world.

Use `wasi-virt`-based VFS to store Python code

Currently, componentize-py "cheats" by assuming all required Python code will be loaded during the component pre-init step, and thus CPython won't need access to any of those files at runtime. This works surprisingly well, but will certainly break for more dynamic apps.

We should probably build a VFS by default, but also offer an option to use the "cheat" if desired.

Matrix-math example - no world named `matrix-math` in package

Following the matrix-math example in fa666b9, I receive the following error:

componentize-py --wit-path src/componentize-py-fa666b9/wit --world matrix-math componentize app -o matrix-math.wasm
Traceback (most recent call last):
  File "/usr/local/bin/componentize-py", line 8, in <module>
    sys.exit(script())
AssertionError: no world named `matrix-math` in package

I attempted to fully qualify the world using its package name, but that didn't help:

componentize-py --wit-path src/componentize-py-fa666b9/wit --world "componentize-py:init/matrix-math" componentize app -o matrix-math.wasm
Traceback (most recent call last):
  File "/usr/local/bin/componentize-py", line 8, in <module>
    sys.exit(script())
AssertionError: unknown package `componentize-py:init`

I'm running the example inside a python:3.10.14-slim container.

Matrix-math example - ImportError...

Following the example of matrix-math fd5b49c I get the following error when running componentize-py -d ../../wit -w matrix-math componentize app -o matrix-math.wasm:


warning: site-packages directory not found under /home/jes/gitrepositories/wasmcloud_test/my-py-component-wasmtime16-0-0/myenv/lib
Traceback (most recent call last):
  File "/home/jes/.local/bin/componentize-py", line 8, in <module>
    sys.exit(script())
             ^^^^^^^^
AssertionError: Traceback (most recent call last):
  File "/0/app.py", line 5, in <module>
    import numpy
  File "/0/numpy/__init__.py", line 157, in <module>
    from . import random
  File "/0/numpy/random/__init__.py", line 180, in <module>
    from . import _pickle
  File "/0/numpy/random/_pickle.py", line 1, in <module>
    from .mtrand import RandomState
ImportError: dynamic module does not define module export function (PyInit_mtrand)


Caused by:
    ImportError: dynamic module does not define module export function (PyInit_mtrand)

I am using python 3.12 and have created an enviroment using virtualenv
I am more experienced in Rust than python and have been unable to figure out if it is an error on my side or with the example.

feature: add zlib to cpython build

Some packages (in particular, aiohttp) require CPython to be built with zlib enabled. I believe this is known work but I wanted to create a tracking issue here for it. (I took a stab at this via building in wasi-wheels but bounced off the problem ๐Ÿ˜…)

Access is denied

When running componentize-py 0.4.0 on Windows for the demo I get the following error:

AssertionError: Access is denied. (os error 5)

Any ideas on where this error is coming from? It's extremely vague and I'm not sure how to determine which part of the code is generating this error

Question: Current types limitation

Hello guys,

First, this is great work, congrats!

I have been playing around and created a custom (but simple) .wit, implemented in Python a function defined in that WIT file, built the .wasm module and then I tried to call the function from a Rust application.

I was able to provide a custom record type as parameter of the function and the component built succeeded (I mean, I generated the .wasm file correctly). However, using either the wasmer or wasmtime crates, the supported set of types you can use in the call function as Value/Val is very small. Ref: https://docs.rs/wasmtime/latest/wasmtime/enum.Val.html

So my question is, am I missing something or it is actually not possible to use a custom record right now as parameter or return type of a function?

I know this is probably not related to the componentize-py tool but a more general question, but I am not sure which would be the best place to ask.

Expand CI test coverage

We should run all the tests on Mac and Windows -- not just Linux. Also, we should test the examples to ensure they keep working.

Create more actionable error for disallowed name for a Python module

Error

I ran into the following error:

$ componentize-py --wit-path add.wit --world example componentize example -o add.wasm
Traceback (most recent call last):
  File "/opt/homebrew/bin/componentize-py", line 8, in <module>
    sys.exit(script())
             ^^^^^^^^
AssertionError:

Caused by:
    TypeError: Can't instantiate abstract class Example without an implementation for abstract method 'add'

Repro

My set up was the following:

// WIT
package example:component;

world example {
    export add: func(x: s32, y: s32) -> s32;
}

App:

$ cat<<EOT >> example.py
import example

class Example(example.Example):
    def add(self, x: int, y: int) -> int:
        return x + y
EOT

Now when i go to componentize I get the TypeError:

$ componentize-py --wit-path /path/to/examples/example-host/add.wit --world example componentize example -o add.wasm
Traceback (most recent call last):
...

Workaround

@dicej pointed out that the issue is that you cannot name the Python module (example.py) the same as the world name (example), since componentize-py will generate a module using the world name and only use one of them. Renaming example.py to app.py resolved the issue.

The error thrown should be more descriptive and actionable. It was not clear from the TypeError how to resolve the issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.