bytecodealliance / componentize-py Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Hello guys,
First, this is great work, congrats!
I have been playing around and created a custom (but simple) .wit
, implemented in Python a function defined in that WIT file, built the .wasm
module and then I tried to call the function from a Rust application.
I was able to provide a custom record
type as parameter of the function and the component built succeeded (I mean, I generated the .wasm
file correctly). However, using either the wasmer
or wasmtime
crates, the supported set of types you can use in the call
function as Value
/Val
is very small. Ref: https://docs.rs/wasmtime/latest/wasmtime/enum.Val.html
So my question is, am I missing something or it is actually not possible to use a custom record
right now as parameter or return type of a function?
I know this is probably not related to the componentize-py
tool but a more general question, but I am not sure which would be the best place to ask.
Per https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md#package-declaration, if the WIT interface has an interface, it needs to be encoded in the component type identifier for that interface.
I'm trying to build the repo from source following the instructions in CONTRIBUTING.md and it works, including running cargo run -- --help
and also the bindings
command but when I try to run componentize
I get the following error:
> cargo run --release -- -d hello.wit -w hello componentize app -o app.wasm
warning: [email protected]: using seed f0dd8d4b7597bc894379cc4d111aed752baa62cfc70fb22e8d4bb2810a9ade4b (set COMPONENTIZE_PY_TEST_SEED env var to override)
warning: [email protected]: using count 10 (set COMPONENTIZE_PY_TEST_COUNT env var to override)
warning: field `package` is never read
--> src/summary.rs:76:9
|
74 | pub struct MyInterface<'a> {
| ----------- field in this struct
75 | pub id: InterfaceId,
76 | pub package: Option<PackageName<'a>>,
| ^^^^^^^
|
= note: `MyInterface` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis
= note: `#[warn(dead_code)]` on by default
warning: `componentize-py` (lib) generated 1 warning
Finished `release` profile [optimized] target(s) in 0.26s
Running `target/release/componentize-py -d hello.wit -w hello componentize app -o app.wasm`
Error: unresolved symbol(s):
libcomponentize_py_runtime.so needs _PyUnicode_Ready (function [I32] -> [I32])
libcomponentize_py_runtime.so needs _CLOCK_PROCESS_CPUTIME_ID (global I32)
libcomponentize_py_runtime.so needs _CLOCK_THREAD_CPUTIME_ID (global I32)
When I build the package with maturin and install it I see the same error but the version installed using pip works in the same virtualenv.
I'm on Arch linux with Python 3.11.8 and the latest nightly for rust stuff.
The current Python binding generator predates the concept of WIT packages, and we need to update it. Otherwise, importing or exporting two interfaces with the same name but different packages will result in a clash.
Per https://docs.python.org/3/library/stdtypes.html#typecontextmanager, the idiomatic way to represent resource types is as context managers, each with an __exit__
method that drops the resource. This will allow them to be used in with
statements.
Currently, if Python code passes the wrong number or types of parameters to the host, or returns a result of an unexpected type, componentize-py
generally reacts by panicking and trapping. That's probably unavoidable in most cases, but it should at least print an informative and actionable diagnostic prior to panicking.
For example, when targeting the wasi:cli command world:
from hello import exports
from time import sleep
class Run(exports.Run):
def run(self):
print("Hello, world!")
sleep(0.00001)
The above prints "Hello, world!" and then seems to sleep forever.
Analogous to bytecodealliance/wasmtime-py#171, we should escape any identifier names which match Python keywords by appending underscores to them.
I'm looking to get the example in the readme working. I have the component wasm compiled via this command:
$ componentize-py -d hello.wit -w hello componentize app -o app.wasm
Now I want to run this via wasmtime
but that gives me this error:
$ wasmtime run --wasm component-model app.wasm
Error: failed to run main module `app.wasm`
Caused by:
exported instance `wasi:cli/[email protected]` not present
I thought I could convert this to a core module with wasm-tools
but that gives me a different error:
$ wasm-tools component new -v app.wasm -o out.wasm --adapt wasi_snapshot_preview1.reactor.wasm
error: failed to encode a component from module
Caused by:
0: unknown table 2: exported table index out of bounds (at offset 0xb)
Ideally I'd like to take Python and covert it into wasm much like how it's done in Javy (https://github.com/bytecodealliance/javy) where the output of javy compile
is code that I can execute with wasmtime <output>.wasm
.
Thanks for any help or ideas! =)
I'm attempting to port the CompontenizeJS example to compotentize-py.
package local:hello;
world hello {
export hello: func(name: string) -> string;
}
componentize-py -d hello.wit -w hello bindings .
import hello
class Hello(hello.Hello):
def hello(self, name) -> str:
return f"Hello, {name}, from Python!"
Trying to run the component fails
componentize-py -d hello.wit -w hello componentize app -o hello.component.wasm
cargo build --release
./target/release/wasmtime-test
Error: import `wasi:cli/[email protected]` has the wrong type
Caused by:
0: instance export `get-environment` has the wrong type
1: expected func found nothing
make: *** [py-hello] Error 1
Currently, componentize-py
generates no code for type aliases; instead, it just uses the dealiased types directly. Instead, we should generate assignment expressions for each alias and use the alias in all the same places (e.g. function parameter types, etc.) that the original WIT files used them.
Hi, After following the instructions in README.md to set up the component and execute it using wasmtime
, the application fails to start with the following error message:
Error: failed to run main module `app.wasm`
Caused by:
exported instance `wasi:cli/[email protected]` not present
Code is here
Some packages (in particular, aiohttp
) require CPython to be built with zlib enabled. I believe this is known work but I wanted to create a tracking issue here for it. (I took a stab at this via building in wasi-wheels
but bounced off the problem ๐
)
I ran into the following issue:
thread '<unnamed>' panicked at 'internal error: entered unreachable code', src/lib.rs:450:35
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/Users/eliabieri/Library/Caches/pypoetry/virtualenvs/widget-4qY9vehV-py3.11/bin/componentize-py", line 8, in <module>
sys.exit(script())
^^^^^^^^
pyo3_runtime.PanicException: internal error: entered unreachable code
Prerequisites: Poetry
git clone --recurse-submodules [email protected]:eliabieri/wg_display_widget_py.git
poetry install
poetry run componentize-py --wit-path wg_display_widget_wit/wit --world widget bindings .
poetry run componentize-py --wit-path wg_display_widget_wit/wit --world widget componentize --output widget.wasm --python-path widget widget
The pre-initialization process currently involves an exported anonymous interface and types (defined in init.wit). That should be stripped out of the final component since it's not part of the application's world.
When running componentize-py 0.4.0 on Windows for the demo I get the following error:
AssertionError: Access is denied. (os error 5)
Any ideas on where this error is coming from? It's extremely vague and I'm not sure how to determine which part of the code is generating this error
Following the matrix-math example in fa666b9, I receive the following error:
componentize-py --wit-path src/componentize-py-fa666b9/wit --world matrix-math componentize app -o matrix-math.wasm
Traceback (most recent call last):
File "/usr/local/bin/componentize-py", line 8, in <module>
sys.exit(script())
AssertionError: no world named `matrix-math` in package
I attempted to fully qualify the world using its package name, but that didn't help:
componentize-py --wit-path src/componentize-py-fa666b9/wit --world "componentize-py:init/matrix-math" componentize app -o matrix-math.wasm
Traceback (most recent call last):
File "/usr/local/bin/componentize-py", line 8, in <module>
sys.exit(script())
AssertionError: unknown package `componentize-py:init`
I'm running the example inside a python:3.10.14-slim
container.
We're currently using component-init
to snapshot the state of an initialized app, and the initialization process may involve getting random numbers from the host and using them to seed one or more PRNGs. That could pose a security risk if we don't force those PRNGs to be reseeded on resume, so we should find these cases in wasi-libc and/or CPython. This could also be an issue in Python code and/or native extensions, but it's not clear how we would address that; we may need to leave that up to the application programmer.
Per yesterday's Python guest tooling meeting, we'd like componentize-py
to download any dependencies found in the application's pyproject.toml
(including transitive dependencies), placing them in a temporary directory for use while building the component.
Similar to micropip, componentize-py
should grab any pure Python packages from PyPI, only looking to an alternative repo for packages with native extensions. We (i.e. the Bytecode Alliance) will need to maintain this repo and populate it with WASI builds of popular projects until WASI wheels are standardized and published upstream to PyPI.
To start with, componentize-py
could shell out to e.g. pip --install --platform wasi-sdk-22 --abi cp311 --root <tmpdir>
, but eventually we'd like to make it fully self-contained (i.e. not require Python or pip
to be installed).
This would dramatically reduce binary sizes in cases where the host can provide these libraries.
This should include a CONTRIBUTING.md
file, plus an ARCHITECTURE.md
describing the high-level architecture of the project. In addition, each source file needs doc and implementation comments.
I'd like to execute code using the sandbox example, but am running into an issue. I want to allow exec'd code to import packages, but if I don't import packages in the host script, the guest code won't run.
# Guest code
import json
print(json.dumps({"message": "Hello"}))
# Works
import sys
import json
from command import exports
class Run(exports.Run):
def run(self):
with open(sys.argv[1]) as f:
code = f.read()
exec(code)
# Fails
import sys
from command import exports
class Run(exports.Run):
def run(self):
with open(sys.argv[1]) as f:
code = f.read()
exec(code)
Attempting an example for the first time, and it looks like there's a typo in the example code.
I run componentize-py -d ../../wit -w wasi:http/[email protected] componentize app -o http.wasm
and get this traceback:
Traceback (most recent call last):
File "/home/daedalus/src/wasm-experiments/.venv/bin/componentize-py", line 8, in <module>
sys.exit(script())
AssertionError: Traceback (most recent call last):
File "/0/app.py", line 10, in <module>
import poll_loop
File "/bundled/poll_loop.py", line 17, in <module>
from proxy.imports.streams import StreamErrorClosed, InputStream
ImportError: cannot import name 'StreamErrorClosed' from 'proxy.imports.streams' (/world/proxy/imports/streams.py). Did you mean: 'StreamError_Closed'?
It looks like bundled/poll_loop.py
does indeed try to import and use StreamErrorClosed
instead of StreamError_Closed
, but when I try to make the change locally (so I can maybe file a PR), it doesn't appear to fix the error.
I clearly don't understand how Python imports work with wasmtime and/or componentize-py. What am I missing? Is it pulling in the import from another path or a prebuilt module somewhere?
No rush, just wanted to start learning about wasm and coming from my background with Python seemed a reasonable entry point.
In addition to running the test suite, we should dump the app code and generated bindings in a directory and run MyPy on them, asserting that no errors are found. This will help us catch any regressions in the type annotations,
Following the example of matrix-math fd5b49c I get the following error when running componentize-py -d ../../wit -w matrix-math componentize app -o matrix-math.wasm
:
warning: site-packages directory not found under /home/jes/gitrepositories/wasmcloud_test/my-py-component-wasmtime16-0-0/myenv/lib
Traceback (most recent call last):
File "/home/jes/.local/bin/componentize-py", line 8, in <module>
sys.exit(script())
^^^^^^^^
AssertionError: Traceback (most recent call last):
File "/0/app.py", line 5, in <module>
import numpy
File "/0/numpy/__init__.py", line 157, in <module>
from . import random
File "/0/numpy/random/__init__.py", line 180, in <module>
from . import _pickle
File "/0/numpy/random/_pickle.py", line 1, in <module>
from .mtrand import RandomState
ImportError: dynamic module does not define module export function (PyInit_mtrand)
Caused by:
ImportError: dynamic module does not define module export function (PyInit_mtrand)
I am using python 3.12 and have created an enviroment using virtualenv
I am more experienced in Rust than python and have been unable to figure out if it is an error on my side or with the example.
Implement support for WIT resources, including the canonical ABI requirements. For reference, see wit-bindgen's Rust and C implementations.
I ran into the following error:
$ componentize-py --wit-path add.wit --world example componentize example -o add.wasm
Traceback (most recent call last):
File "/opt/homebrew/bin/componentize-py", line 8, in <module>
sys.exit(script())
^^^^^^^^
AssertionError:
Caused by:
TypeError: Can't instantiate abstract class Example without an implementation for abstract method 'add'
My set up was the following:
// WIT
package example:component;
world example {
export add: func(x: s32, y: s32) -> s32;
}
App:
$ cat<<EOT >> example.py
import example
class Example(example.Example):
def add(self, x: int, y: int) -> int:
return x + y
EOT
Now when i go to componentize I get the TypeError
:
$ componentize-py --wit-path /path/to/examples/example-host/add.wit --world example componentize example -o add.wasm
Traceback (most recent call last):
...
@dicej pointed out that the issue is that you cannot name the Python module (example.py
) the same as the world name (example
), since componentize-py
will generate a module using the world name and only use one of them. Renaming example.py
to app.py
resolved the issue.
The error thrown should be more descriptive and actionable. It was not clear from the TypeError
how to resolve the issue.
I think this is actually a bug but it may not be one that idiomatic wit files would trigger so it may not be an important issue but I'll post it anyway just in case.
When records are defined in a world along side functions which use the record type, the functions are generated first in the binding's init.py file so that type checking fails due to the undefined type.
For example, this wit file will not produce usable bindings:
package example:broken;
world broken-world {
record a-record {
field: s32
}
import get-a-record: func() -> a-record;
export do-stuff: func() -> a-record;
}
My naive fix was to stick from __future__ import annotations
at the top of the __init__.py
file and that does work. But then I realized that the idiomatic thing to do seems to be to put type definitions in a separate types interface and import from there. Doing that produces valid bindings without having to change componentize_py at all.
We should run all the tests on Mac and Windows -- not just Linux. Also, we should test the examples to ensure they keep working.
I have a Rust project that loads and executes wasm components using wasmtime
.
The wasmtime
usage can be seen here.
If I try to instantiate a component generated using componentize-py
, I get an error concerning a type mismatch with a WASI type
import `wasi:clocks/wall-clock` has the wrong type
componentiue-py
?
wasmtime
?Apologies if these questions should be clear from the documentation. I was just not able to figure these out, even after trying for some time.
Currently, componentize-py
"cheats" by assuming all required Python code will be loaded during the component pre-init step, and thus CPython won't need access to any of those files at runtime. This works surprisingly well, but will certainly break for more dynamic apps.
We should probably build a VFS by default, but also offer an option to use the "cheat" if desired.
Python does not currently have a way to represent the types of errors/exceptions which might be raised by a function in a typed function signature, so the next best thing is to list them in the docstring (e.g. following any docs derived from WIT doc comments).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.