wasmx / wasm-chisel

Some useful utilities to transform WebAssembly binaries.

wasm-chisel

Some useful utilities to transform WebAssembly binaries, most importantly for WebAssembly used in a deterministic / blockchain context, such as with ewasm.

Library

remapimports

Provide a list of imports (with namespace and name) and replace them with a new set of namespace and name pairs.

This can be very useful together with compilers, which do not support the specification of a namespace in imports yet. As of writing mid-2018, that includes pretty much every compiler (one exception is AssemblyScript).

It supports the same presets as verifyimports.

trimexports

Removes all exports, but the ones specified.

This comes with some presets:

• ewasm: keeps main and exported memory
• pwasm: keeps _call

trimstartfunc

Remove start function.

This comes with the following preset:

• ewasm: removes start function if present

verifyimports

Verifies that the module's imports are compliant with the provided import interface. Can be set to require the existence of the entire import set, or just the validity of existing imports with matching identifiers. Can be set to allow or prohibit unlisted additional imports.

The following presets are provided:

• ewasm: Verifies the ewasm EEI. Disallows unlisted imports, and does not require that the entire interface be imported.
• debug: Debug utilities for ewasm.
• bignum: Big-number library for ewasm.
• eth2: Verifies imports according to Scout.

verifyexports

Verifies that the module's exports are compliant with the provided export interface. Can be set to allow or prohibit unlisted additional exports.

The following presets are provided:

• ewasm: Verifies that the main function and memory is exported. Disallows any unlisted exports.

dropsection

Removes selected sections from the module.

deployer

Wraps module into an ewasm-compatible constructor. It has two presets:

• memory: wrap the module as a pre-defined memory section
• customsection: include the module as a custom section

repack

Re-serializes the module. It will drop any unknown (custom) sections.

remapstart

If there is a start section, export it as main (replacing any pre-existing main export) and remove the start section

snip

Wraps wasm-snip and turns on removing Rust formatting and debugging from wasm.

dropnames

Drops the NamesSection if present.

CLI

chisel is available as a command line tool. It supports two usage patterns, config-driven and unix-style, also known as oneliner.

It uses features implemented in the library as well in wasm-gc and wasm-utils. It comes with a configuration file chisel.yml.

chisel run: searches for chisel.yml in the current directory, if not specified otherwise using the flag -c. Runs the modules specified in the configuration, outputs a new file if any changes were made by translator or creator modules, and prints a brief report of each module's results.

chisel: Invokes unix-style mode. It requires the flags --modules and --config.

• --modules takes a comma-separated list of modules to be invoked. An example of a valid module list is: --modules remapimports,verifyimports

• --config takes a comma separated list of key-value options for the modules to be invoked, in the form module.option=value. An example of a valid configuration is: --config remapimports.preset=ewasm,verifyimports.preset=ewasm

Configuration file

The configuration file starts with a ruleset entry, where the name can be anything. Inside the ruleset are its options.

The only required field is file, which specifies the path to the Wasm binary to be chiseled.

Optionally, one may also specified an output file through the output option.

It is important to note that the configuration parsing will not work if all the rules are prepended with a hyphen. Please avoid this until the configuration parser is generalized.

ewasm:
  file: "target/wasm32-unknown-unknown/release/sentinel.wasm"
  output: "out.wasm"
  remapimports:
    preset: "ewasm"

sentinel.rs

TBA

Changelog

A changelog is available.

Maintainers

• Alex Beregszaszi @axic
• Jake Lang @jakelang

License

Apache 2.0.

wasm-chisel's Issues

Locally implemented wasm-snip functionality

a la wasm-snip

Key-value configuration for modules

ModulePreset is extremely inflexible. Instead, modules should be configured in a more ad-hoc manner by way of a key-value mapping.

RemapImports is missing a translation for various EEI methods

Implement metering injection

Module to check for floating point operations

Configuration file

From the README:

The configuration file starts with a ruleset entry, where the name can be anything. Inside the ruleset are its options.

ewasm:
  - file: "target/wasm32-unknown-unknown/release/sentinel.wasm"
  - remapimports:
    - style: ewasm
  - deployer

Add WASI import list to verifyimports/renameimports

VerifyImports fails on code that has been transformed by RemapImports

For example:

(module
  (memory 1)
  (type (;0;) (func (result i64)))
  (import "ethereum" "getGasLeft" (type 0))
  (func $main)
  (export "main" (func $main))
  (export "memory" (memory 0))
)

Translator which replaces known functions with imports

This in practice is the inverse of #38.

Basically searches for a known function, removes that and includes an import instead.

Change ModuleTranslator to return a module rather than operate on a mutable reference

From #33 (comment)

chisel: generalize YAML config parser

At the moment, everything has to be a hash map. This means a module cannot be used twice because its hashes will collide.

Time to generalize this so a list can be used, etc.

Cleanup remapimports to have a nicer structure

Debug logging

CLI

The original proposal for CLI was only have a configuration format.

This briefly described in the README: https://github.com/wasmx/wasm-chisel#configuration-file-wip

@jakelang has raised that it would make sense to have a stand alone tool too.

I think we could do something like this, in line with other tools:

• chisel carve (with chisel run for the old-school folks) which looks for a configuration file and executes it
• chisel carve <module> {params} <filename>

Support optimisations from binaryen

VerifyExports returns false when a data section is present, even if exports are correct

Translator to drop sections

Deployer: fix memory size

The deployer has a fixed 1 page memory size (this applies to both the memory and the custom section deployre). Likely it will fail with larger contacts. This needs to be fixed and tested for.

verifyimports seem to fail on ewasm-rust-api generated code

All of the precompiles fail with something similar:

$ target/debug/chisel sentinel ewasm_precompile_bls12pairing.wasm 
========== SENTINEL ==========
thread 'main' panicked at 'index out of bounds: the len is 5 but the index is 5', libchisel/src/verifyimports.rs:507:38
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Support reading wat (add an option to online+config)

Implement verifyimports

Use CustomSection::new

paritytech/parity-wasm#244 added it, released in 0.35.2

Merge the EEI import list between verifyimports and renameimports

Cargo Integration

It seems that integrating chisel with rust projects requires either a makefile or manual invocation, which isn't great ux. Perhaps a cargo subcommand or plugin would make this much easier.
cc @axic

Confirm whether updating names section is neded after changing exports

See #99 (comment)

Translator which replaces known imports with snippets

From: https://github.com/ewasm/ewasm-cleanup#libc-methods

Goal is to have a translator which takes a wasm module (called the "helper") and a list of imports. The current module is searched for the list of imports. If found, the import is removed and the function from the helper is copied in.

The first two to start with:

• memcpy(i32, i32, i32) -> i32 from https://github.com/ewasm/ewasm-cleanup/blob/master/memcpy.wast
• memset(i32, i32, i32) from https://github.com/ewasm/ewasm-cleanup/blob/master/memset.wast

Rework README to be nicer

• Move CLI usage to the top.
• Update module descriptions.
• Update links to tools using chisel.

Make RemapImports more generic by removing the requirement of a prefix

Add Codecov

cc @axic

Statistics interface for modules

The user may want to retrieve more information about a module's execution.
This can be done through a special ChiselResult type, or a trait to call on an executed module.

Translator for wasm-bulk-memory

Introduce two modules which translate:

1. from regular wasm to bulk-memory
2. from bulk-memory to regular wasm

Overview: https://github.com/WebAssembly/bulk-memory-operations/blob/master/proposals/bulk-memory-operations/Overview.md

As an example, replace all memcpy's with the following function:

(func (param $dst i32) (param $src i32) (param $size i32) (result i32)
  get_local $dst
  get_local $src
  get_local $size
  memory.copy
  get_local $dst)

Or for the reverse direction: replace the use of memory.copy with memcpy.

Expose walrus::gc as a helper

From #24 (comment)

WASM validator module

This should do semantic validation of the module according to the Wasm specification.

Must also first check how much of that is done by pwasm.

error[E0463]: can't find crate for `chisel`

When I run my rust contract I got error forchisel
Here is my code. TIA

#![feature(proc_macro, wasm_custom_section, wasm_import_module)]
extern crate chisel;
extern crate ewasm_api;

#[no_mangle]
pub extern "C" fn add_one(name :&str) {
    println!("hello")
}

Here's log

cargo build
   Compiling rust_contract v0.1.0 (/home/achala/Codes/kip-gitlab/kide/templates/empty_rust)
error[E0463]: can't find crate for `chisel`
 --> src/lib.rs:2:1
  |
2 | extern crate chisel;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `rust_contract`.

To learn more, run the command again with --verbose.

needs a default configuration file

The docs are wrong about the CLI tool.

It comes with a configuration file chisel.yml.

Add wrapper for wasm-gc

While wasm-gc was discontinued, due to Rust having LTO turned on, it is still useful for other languages besides Rust.

https://github.com/alexcrichton/wasm-gc

Module to replace a function with an import

Requested by cdetrio

Example:

chisel replaceFuncWithImport --funcName "$memcpy" --import "env.$memcpy"

input:
  * a wasm module with a named function (i.e. a function body in the wasm code)
output:
  * the wasm module where the named function is an import instead, and the function body is deleted

Introduce IR for analysis

Or at least helpers with some data structures.

A good start would be doing a CFG (control flow graph) or at least a primitive version which shows dependencies of functions.

This can be used in two ways in the short term:

• finding unused functions
• in "atomic metering" to find the leaf nodes to start metering on

feature for adding wat functions to a wasm file

Suppose you have a wasm file like this:

(module $chiselExample
  (type $t2 (func (param i32 i32) (result i32)))
  (import "ewasm" "addmod" (func $addmod256 (type $t2)))
  (export "main" (func $main))
  (func $main (result i32)
    (local $pointx i32) (local $pointy i32)

    (i64.store (i32.const 0) (i64.const 7771015739068247266)) ;; point.x
    (i64.store (i32.const 8) (i64.const 15978182637586970136))
    (i64.store (i32.const 16) (i64.const 1690519934368996064))
    (i64.store (i32.const 24) (i64.const 630815934106406715))
    (local.set $pointx (i32.const 0))

    (i64.store (i32.const 32) (i64.const 13209253545858841660)) ;; point.y
    (i64.store (i32.const 40) (i64.const 1023134550666483074))
    (i64.store (i32.const 48) (i64.const 13677454995611397236))
    (i64.store (i32.const 56) (i64.const 249532354227923480))
    (local.set $pointy (i32.const 32))

    (call $addmod256
      (local.get $pointx) (local.get $pointy))

    (return)
  )
  (memory $memory (export "memory") 50)
)

Above, the $addmod256 function is an imported host function. But suppose you also have a wasm implementation of that function:

(module $addmodForChisel
  (type $t2 (func (param i32 i32) (result i32)))
  (func $addmod256 (type $t2)
    (param $pointx i32) (param $pointy i32)
    (result $resultx i32)
    ;; bunch of hand-written wast code...
    (return)
  )
)

If you'd like to use the wasm function instead of the host function, it would be useful if chisel could paste the code for $addmod256 from the second file into the first file, producing a wasm module like this:

(module $chiselExample
  (type $t2 (func (param i32 i32) (result i32)))
  (export "main" (func $main))
  (func $main (result i32)
    (local $pointx i32) (local $pointy i32)

    (i64.store (i32.const 0) (i64.const 7771015739068247266)) ;; point.x
    (i64.store (i32.const 8) (i64.const 15978182637586970136))
    (i64.store (i32.const 16) (i64.const 1690519934368996064))
    (i64.store (i32.const 24) (i64.const 630815934106406715))
    (local.set $pointx (i32.const 0))

    (i64.store (i32.const 32) (i64.const 13209253545858841660)) ;; point.y
    (i64.store (i32.const 40) (i64.const 1023134550666483074))
    (i64.store (i32.const 48) (i64.const 13677454995611397236))
    (i64.store (i32.const 56) (i64.const 249532354227923480))
    (local.set $pointy (i32.const 32))

    (call $addmod256
      (local.get $pointx) (local.get $pointy))

    (return)
  )

  (func $addmod256 (type $t2)
    (param $pointx i32) (param $pointy i32)
    (result $resultx i32)
    ;; bunch of hand-written wast code...
    (return)
  )

  (memory $memory (export "memory") 50)
)

Remove compiler warnings

I think its good if we get rid of the warnings generated by the compiler

warning: unused import: `parity_wasm::builder::*`
 --> libchisel/src/dropsection.rs:3:5
  |
3 | use parity_wasm::builder::*;
  |     ^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: #[warn(unused_imports)] on by default

warning: unused import: `ValueType`
 --> libchisel/src/verifyimports.rs:5:82
  |
5 | use parity_wasm::elements::{External, FunctionType, ImportSection, Module, Type, ValueType};
  |                                                                                  ^^^^^^^^^

warning: unused imports: `CodeSection`, `Func`, `FunctionSection`
 --> libchisel/src/depgraph.rs:3:29
  |
3 | use parity_wasm::elements::{CodeSection, Func, FuncBody, FunctionSection, Instruction, Module};
  |                             ^^^^^^^^^^^  ^^^^            ^^^^^^^^^^^^^^^

warning: unused import: `crate::utils`
 --> libchisel/src/deployer.rs:2:5
  |
2 | use crate::utils::*;
  |     ^^^^^^^^^^^^

warning: unreachable pattern
   --> libchisel/src/verifyexports.rs:155:17
    |
155 |                 _ => None,
    |                 ^
    |
    = note: #[warn(unreachable_patterns)] on by default

warning: unreachable pattern
   --> libchisel/src/verifyexports.rs:161:21
    |
161 |                     _ => None,
    |                     ^

warning: unused variable: `module`
  --> libchisel/src/repack.rs:14:33
   |
14 |     fn translate_inplace(&self, module: &mut Module) -> Result<bool, ModuleError> {
   |                                 ^^^^^^ help: consider prefixing with an underscore: `_module`
   |
   = note: #[warn(unused_variables)] on by default

warning: unused variable: `start_section`
 --> libchisel/src/trimstartfunc.rs:8:21
  |
8 |         if let Some(start_section) = module.start_section() {
  |                     ^^^^^^^^^^^^^ help: consider prefixing with an underscore: `_start_section`

warning: unused variable: `module`
  --> libchisel/src/trimstartfunc.rs:31:25
   |
31 |     fn translate(&self, module: &Module) -> Result<Option<Module>, ModuleError> {
   |                         ^^^^^^ help: consider prefixing with an underscore: `_module`

warning: unused variable: `import_section`
   --> libchisel/src/verifyexports.rs:164:25
    |
164 |             (None, Some(import_section)) => None,
    |                         ^^^^^^^^^^^^^^ help: consider prefixing with an underscore: `_import_section`

warning: unused variable: `import_section`
   --> libchisel/src/verifyimports.rs:277:9
    |
277 |     let import_section = module.import_section().expect("No function section found");
    |         ^^^^^^^^^^^^^^ help: consider prefixing with an underscore: `_import_section`

warning: variable does not need to be mutable
   --> libchisel/src/verifyimports.rs:105:21
    |
105 |                 let mut checklist: Vec<ImportStatus> = self
    |                     ----^^^^^^^^^
    |                     |
    |                     help: remove this `mut`
    |
    = note: #[warn(unused_mut)] on by default

warning: variable does not need to be mutable
  --> libchisel/src/utils.rs:14:13
   |
14 |         let mut module = self
   |             ----^^^^^^
   |             |
   |             help: remove this `mut`

warning: private trait `depgraph::DepGraphManager` in public interface (error E0445)
  --> libchisel/src/depgraph.rs:27:1
   |
27 | / pub trait DepGraphBuilder: DepGraphManager {
28 | |     /// Builds the dependency graph.
29 | |     fn build(module: &Module, entry_idx: u32) -> Result<Self, ()>
30 | |     where
31 | |         Self: std::marker::Sized;
32 | | }
   | |_^
   |
   = note: #[warn(private_in_public)] on by default
   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
   = note: for more information, see issue #34537 <https://github.com/rust-lang/rust/issues/34537>

warning: method is never used: `insert`
   --> libchisel/src/remapimports.rs:177:5
    |
177 |     fn insert(&mut self, from_module: &str, from_field: &str, to_module: &str, to_field: &str) {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: #[warn(dead_code)] on by default

warning: method is never used: `edgecount`
  --> libchisel/src/depgraph.rs:41:5
   |
41 |     pub fn edgecount(&self) -> usize {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   Compiling serde_derive v1.0.92
   Compiling serde_yaml v0.8.9
   Compiling chisel v0.4.0 (/private/tmp/wasm/wasm-chisel/chisel)
warning: variable does not need to be mutable
  --> chisel/src/main.rs:76:24
   |
76 |             for (name, mut config) in rules.iter().filter(|(left, right)| match (left, right) {
   |                        ----^^^^^^
   |                        |
   |                        help: remove this `mut`
   |
   = note: #[warn(unused_mut)] on by default

Improve rustdoc

Introduce code metrics and optimise guided by them

This in practice suggest to use a "genetic algorithm" guided by the metrics (fitness function).

One metric we could implement is execution time on an interpreter and try to optimise code for that. The optimisation passes form binaryen (and others) can be used.

Translator for start function (remapstart)

1. Rename the function pointed to by the start section to main. If there is another function named main, rename it (prefix it with _) to avoid conflict – if there this conflicts with another function, keep adding underscore prefixes until it won't.

2. Remove the start section. (See dropstartfunction.)

Translator to remove unreferenced functions

Replace wabt-rs dependency with wat/wast

See:

• https://crates.io/crates/wast
• https://crates.io/crates/wat

YAML need to support multiple top level keys

This doesn't seem to work (e.g. only one of them is run, not sure which):

blake2:
  file: "target/wasm32-unknown-unknown/release/ewasm_precompile_blake2.wasm"
  remapimports:
    preset: "ewasm"
  trimexports:
    preset: "ewasm"

bls12pairing:
  file: "target/wasm32-unknown-unknown/release/ewasm_precompile_bls12pairing.wasm"
  remapimports:
    preset: "ewasm"
  trimexports:
    preset: "ewasm"

Translator to drop unused imports

Add translator which drops unused imports.