Coder Social home page Coder Social logo

lincoln-lm / numba-wasm-example Goto Github PK

View Code? Open in Web Editor NEW
9.0 2.0 2.0 214 KB

Example of compiling numba functions to wasm for use in JS or alongside pyodide

Home Page: https://lincoln-lm.github.io/numba-wasm-example/

License: GNU General Public License v3.0

HTML 13.20% Python 86.80%
llvm numba python wasm webassembly pyodide

numba-wasm-example's Introduction

numba-wasm-example

Example of compiling numba functions to wasm for use in JS or alongside pyodide

This repository contains two python packages set up with poetry:

numba_wasm - A very basic library that provides compilation to and interfacing with WebAssembly.

The library provides a custom @njit_wasm decorator that handles both compiling to wasm and interfacing between pyodide and the wasm compiled functions.

When called from a regular interpreter, the decorator acts mostly the same as the traditional numba.njit, with the exception that it infers the function signature from type annotations rather than relying on it being explicitly declared. This allows easier testing of compiled functions as you do not need to compile to wasm and test in a pyodide environment to test simple things like logic.

When called from a regular interpreter with the "BUILD_WASM_IR" environment variable set (ex. build_numba_functions.py), the decorator compiles to LLVM IR targetted at a 32-bit memory system (with the help of some patches to trick numba) in order to allow emscripten to compile it to wasm32.

When called from a pyodide interpreter, the decorator ignores the actual contents of the function and instead interfaces with the appropriate compiled WebAssembly function, converting inputs to the format the compiled code expects (ex. arrays are converted into pointers when appropriate).

The combination of these three different functionalities allows the same module that contains your code to be used for local testing, compilation, and use within pyodide itself.

example_module - An incredibly simple example library containing basic functions to be compiled to WebAssembly via numba_wasm.

The library provides basic functions marked with the @njit_wasm decorator to test arithmetic and array creation/modification

It also contains the aforementoned build_numba_functions.py script used to generate and export LLVM IR for the functions.

This IR can then be compiled with emscripten as a wasm side module to be loaded directly into the browser via pyodide's interface as is done in the example page (src).

With the compiled wasm side module loaded, using the functions is as simple as loading and importing the wheels for both numba_wasm and example_module into the pyodide interpreter and directly calling the functions (ex. example_module.example.square(2.0)).

Very minimal (console) example page: https://lincoln-lm.github.io/numba-wasm-example/ (src)

numba-wasm-example's People

Contributors

lincoln-lm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

numba-wasm-example's Issues

Return type unspecified should infer numba.void

I think this makes logical sense, but perhaps introduces room for error when an exception isn't explicitly raised when return type isn't specified. Though, it's not like the currently raised exception is readable enough to be super useable.

error: clang frontend command failed due to signal

Thank you for this useful work.
I strictly follow your compliation instructions in workflow.yml, everything works fine (I can also run this: emcc ./numba/numba/core/runtime/nrt.cpp -g -sSIDE_MODULE -o nrt.wasm, but the following one fails), but there is an error when compiling ./example_module/example_module.ll into example_module.wasm:

emcc -v ./example_module/example_module.ll -g -sSIDE_MODULE -s EXPORT_ALL=1 -o example_module.wasm

The error said:

 "D:/emsdk/upstream/bin\clang.exe" --version
 "D:/emsdk/upstream/bin\clang++.exe" -target wasm32-unknown-emscripten -fignore-exceptions -fPIC -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN --sysroot=D:\emsdk\upstream\emscripten\cache\sysroot -Xclang -iwithsysroot/include\fakesdl -Xclang -iwithsysroot/include\compat -v -g3 ./example_module/example_module.ll -c -o C:\Users\Karbo\AppData\Local\Temp\emscripten_temp_d4yrmxdq\example_module_0.oclang version 17.0.0 (https://github.com/llvm/llvm-project 86339ef088e9aac6cf52cf9022d7a5bd144d4c42)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: D:\emsdk\upstream\bin
 (in-process)
 "D:\\emsdk\\upstream\\bin\\clang++.exe" -cc1 -triple wasm32-unknown-emscripten -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name example_module.ll -mrelocation-model pic -pic-level 2 -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-feature +mutable-globals -target-cpu generic -debug-info-kind=constructor -dwarf-version=4 -debugger-tuning=gdb -v "-fcoverage-compilation-dir=E:\\GitRepo\\numba-wasm-example" -resource-dir "D:\\emsdk\\upstream\\lib\\clang\\17" "-fdebug-compilation-dir=E:\\GitRepo\\numba-wasm-example" -ferror-limit 19 -fmessage-length=120 -fvisibility=default -fgnuc-version=4.2.1 -fignore-exceptions -fcolor-diagnostics "-iwithsysroot/include\\fakesdl" "-iwithsysroot/include\\compat" -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o "C:\\Users\\Karbo\\AppData\\Local\\Temp\\emscripten_temp_d4yrmxdq\\example_module_0.o" -x ir ./example_module/example_module.ll
clang -cc1 version 17.0.0 based upon LLVM 17.0.0git default target x86_64-pc-windows-msvc
warning: overriding the module target triple with wasm32-unknown-emscripten [-Woverride-module]
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: D:/emsdk/upstream/bin\\clang++.exe -target wasm32-unknown-emscripten -fignore-exceptions -fPIC -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN --sysroot=D:\\emsdk\\upstream\\emscripten\\cache\\sysroot -Xclang -iwithsysroot/include\\fakesdl -Xclang -iwithsysroot/include\\compat -v -g3 ./example_module/example_module.ll -c -o C:\\Users\\Karbo\\AppData\\Local\\Temp\\emscripten_temp_d4yrmxdq\\example_module_0.o
1.      Code generation
Exception Code: 0x80000003
 #0 0x00007ff6e62f2971 (D:\emsdk\upstream\bin\clang++.exe+0x22971)
 #1 0x00007ff6e67f29e6 (D:\emsdk\upstream\bin\clang++.exe+0x5229e6)
 #2 0x00007ff6e67fb227 (D:\emsdk\upstream\bin\clang++.exe+0x52b227)
 #3 0x00007ff6e64a96d1 (D:\emsdk\upstream\bin\clang++.exe+0x1d96d1)
 #4 0x00007ff6e64a1eb1 (D:\emsdk\upstream\bin\clang++.exe+0x1d1eb1)
 #5 0x00007ff6e70ed9c4 (D:\emsdk\upstream\bin\clang++.exe+0xe1d9c4)
 #6 0x00007ff6e73f4354 (D:\emsdk\upstream\bin\clang++.exe+0x1124354)
 #7 0x00007ff6e733599b (D:\emsdk\upstream\bin\clang++.exe+0x106599b)
 #8 0x00007ff6e673e35d (D:\emsdk\upstream\bin\clang++.exe+0x46e35d)
 #9 0x00007ff6e67e615b (D:\emsdk\upstream\bin\clang++.exe+0x51615b)
#10 0x00007ff6e62dad81 (D:\emsdk\upstream\bin\clang++.exe+0xad81)
#11 0x00007ff6e62d5f43 (D:\emsdk\upstream\bin\clang++.exe+0x5f43)
#12 0x00007ff6e7108bcd (D:\emsdk\upstream\bin\clang++.exe+0xe38bcd)
#13 0x00007ff6e710848b (D:\emsdk\upstream\bin\clang++.exe+0xe3848b)
#14 0x00007ff6e672ad48 (D:\emsdk\upstream\bin\clang++.exe+0x45ad48)
#15 0x00007ff6e672b0de (D:\emsdk\upstream\bin\clang++.exe+0x45b0de)
#16 0x00007ff6e6710c79 (D:\emsdk\upstream\bin\clang++.exe+0x440c79)
#17 0x00007ff6e62d5179 (D:\emsdk\upstream\bin\clang++.exe+0x5179)
#18 0x00007ff6e62ec844 (D:\emsdk\upstream\bin\clang++.exe+0x1c844)
#19 0x00007ff6e9a93ab8 (D:\emsdk\upstream\bin\clang++.exe+0x37c3ab8)
#20 0x00007ffc127e26ad (C:\Windows\System32\KERNEL32.DLL+0x126ad)
#21 0x00007ffc1382aa68 (C:\Windows\SYSTEM32\ntdll.dll+0x5aa68)
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 17.0.0 (https://github.com/llvm/llvm-project 86339ef088e9aac6cf52cf9022d7a5bd144d4c42)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: D:\emsdk\upstream\bin
clang++: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
emcc: error: 'D:/emsdk/upstream/bin\clang++.exe -target wasm32-unknown-emscripten -fignore-exceptions -fPIC -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN --sysroot=D:\emsdk\upstream\emscripten\cache\sysroot -Xclang -iwithsysroot/include\fakesdl -Xclang -iwithsysroot/include\compat -v -g3 ./example_module/example_module.ll -c -o C:\Users\Karbo\AppData\Local\Temp\emscripten_temp_d4yrmxdq\example_module_0.o' failed (returned 1)

I would like to know how to debug such an error, as the error message does not provide much useful information about the cause.


Environment:

  • emcc: 3.1.37 (d02c29f95e9e68f2f50c2b0432c56a671814d5c8)
  • OS: win11 (22H2)

example module link broken

i am trying to find a example for integrating this with pyodide (specifically pygbag but that is just pyodide with some extra packages) but the example module link is broken

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.