Lunatic already has a plugin system that is not well documented or particularly useful in the current state. It started out as a way to dynamically change host functions with WebAssembly modules, but ended up being a tool to modify WebAssembly modules before instantiating them.
I want to use this issue to discuss pros and cons of different approaches to a plugin system and figure out what the best path forward would be. In my grand vision plugins would be an easy way to extend the VM without needing to recompile it, just by adding Wasm modules to it. First of all this means extending the host functions. It could also be a playground for adding new functionality to the VM; we can test it as a plugin behind a feature flag and later provide a native implementation. This could potentially also improve development speed, because plugins can be dynamically loaded and added to environments inside one VM instance. Testing plugins or comparing performance between competing implementations becomes trivial, just spawn different environments, attach different plugins to them and run the same code. You could implement plugins in this case in any language compiling to Wasm, you would not only be limited to rust when extending the VM.
Another use case could be replacing existing functionality with custom implementations. One example of this would be to provide a custom networking layer for the distributed lunatic implementation.
We can go even further here and allow people to load native shared libraries that are exposed as host functions (an idea of @jtenner). Or even allow plugins to modify the loaded Wasm code before JIT compiling it, an approach that our heap profiler plugin is taking.
The most important questions I would like answered here are:
1. Plugins that provide host functions
Originally I wanted lunatic to be just a set of core APIs (networking, filesystem, processes, ...) and every other functionality would be implemented as a plugin in WebAssembly on top of it. Around the same time Wasmtime got support for module-linking and I saw a great opportunity to actually accomplish this. The idea was simple, you could add new host functions to processes by defining them inside WebAssembly plugins. These host functions would provide higher level abstractions on top of the core APIs or even shadow/change the core APIs.
Lunatic would provide a set of "standard" plugins. WASI could be implemented that way, re-exposing lunatic's core filesystem API to guest code through a WASI compatible interface. You could also provide your own plugins by passing them to the lunatic binary: lunatic --plugin my-wasi.wasm ...
. That way you could provide a WASI implementation that actually uses lunatic's networking API and proxies your filesystem read/writes to a network storage.
It would give us an opportunity to keep the core API simple and small, but provide an elegant way to extend it without needing to step out of the WebAssembly space. The developer experience around creating these plugins would be simple, you just import existing host functions and use them inside new host functions that you export.
There are a few drawback to this approach. The module-linking proposal requires each module to be instantiated separately. This means that for each process we are now creating not one WebAssembly instance, but n (where n is the number of plugins we are loading). Process spawning speed is really important for most lunatic use cases and having it progressively slow down as we are extending functionality with plugins was not something I could easily accept. One solution here could be to have the module-linking implementation be lazy and only instantiate if we actually ever use plugin provided host function from a particular process, but the WebAssembly runtime we are using (Wasmtime) doesn't provide such a feature.
A bigger issue is the performance overhead when proxying calls. Let's say someone provides a wasi-socket plugin that builds on top of our core networking API to provide a WASI compatible networking interface. Now writing to the network would require us to first copy all the data from the guest process heap into the wasi-socket plugin instance. If the wasi-socket was building on top of other plugins it would even take more copies, because each layer can just talk to the next one and move data in-between. An alternative to creating all of theses copies would be a much more complex implementation of APIs that allow you to export different memories from different plugins and have a system of core APIs that also take memory "indexes". However, such a system would be much more complicated to work with. You would need to keep track of memory references and slices inside of plugins, on top of the logic you are implementing.
The interface-types proposal is not much of help here as it also assumes copying. From what I understand reading the spec, the Wasm runtime could potentially optimize out this copies, but it would only work between guest code and native host functions, and can't be optimized out between Wasm modules linked together (our case). There are some additions to interface types that could make it work (streams), but it's hard to tell at this stage how exactly everything will fall into place or when this might land.
Lunatic's design shines in I/O heavy workloads and introducing additional overhead is a no-go here. The only way forward would be our custom implementation with memory indexes and then eventually move to interface-types streams once they are ready.
2. Plugins that modify the WebAssembly bytecode
There are some use cases where you want plugins to modify the loaded WebAssembly bytecode. Our heap profiler plugin works that way. It examines the module, looks for functions with names such as malloc, alloc, or similar, and inserts additional call instructions into them that hook up to some host functions that count the memory usage.
One big issue with this approach is that it's really hard to modify WebAssembly modules correctly and depending from which host language you are compiling to WebAssembly the assumptions you are making about the generated bytecode may not hold (e.g. language doesn't have functions with the names malloc, alloc, ...).
The first implementation of this system in lunatic was just giving the whole binary module to the plugin and taking the modified one back. This required each plugin to ship with a whole WebAssembly parser inside and always re-parse the module. Also the order in which the plugins are loaded become significant in this case and can introduce weird issues.
The current implementation parses once the module and exposes hooks to plugins to query the module for functions, modify them or add new ones. I think that this is an ok way forward, it keeps the plugin modules small and we can only expose hooks that are "safe" to use (won't produce incorrect bytecode). It limits the number of ways in which you can modify the module, but this could be a good thing. I spent a lot of time writing Wasm code modification in the first versions of lunatic and it's super easy to produce broken modules.
Even it's super hard to write correct plugins by modifying the Wasm bytecode, I feel like there are always going to be cases where we need it. One good example would be providing polyfills. For some time lunatic used reference types in host functions to manage resources, but many languages don't support reference types on the guest side (rust, c, ...). What we would do is check during loading of the module if there was a mismatch between signatures on the guest and host, if yes we would polyfill the guest with wrappers that save the reference types into a local table and return the index (i32) inside of the table to the guest code that knows how to work with i32 values. If we ever want to introduce host functions that use reference types or interface types again, we will probably need to provide some polyfills for languages that don't understand these "higher level" concepts.
3. Native plugins
This idea is quite simple, but I'm not sure how it would be implemented or if it would even make sense to have it. You would be able to add a native shared library as a plugin (e.g. lunatic --plugin dangerous_stuff.so/dylib/dll ...
). Of course these plugins would be OS and CPU architecture specific, but they would give you the power to call any native functions directly from Wasm modules. You would simply use them like every other host function, just import them by name.
I assume that eventually someone is going to want to hand write some assembly, use a specific peace of hardware or just want to call a library that can't be compiled to WebAssembly yet. This would be the easiest way give them access to it. On the other hand this breaks all security for the current VM instance and all processes inside of it, as the native code can do anything it wants. Once we get distributed lunatic this can be worked around by running a node just for native plugins, that would not have access to the memory space of other instances of the VM, basically isolating any damage the native plugin could do just to a small set of selected processes.
Conclusion
These 3 points should cover almost all use cases. We just need to figure out what's the right balance of power and complexity we want to actually expose to plugin writers. How can we make plugins performant and a joy to write?
I would also love to hear feedback from the community, what do you think? Are there maybe alternative approaches that would be interesting?