wasilibs / nottinygc Goto Github PK

vikas@ xcp (master) $ go version
go version go1.19 linux/amd64
vikas@ xcp (master) $ tinygo build -o /work/out/wasi_wasm/xcp-guard.wasm -gc=custom -tags='custommalloc nottinygc_envoy' -scheduler=none -target=wasi main.go
error: Linking globals named 'runtime.SetFinalizer': symbol multiply defined!
make[1]: *** [Makefile.core.mk:147: out/wasi_wasm/xcp-guard.wasm] Error 1
make: *** [Makefile:58: docker.xcp-guard] Error 2

I am hitting above shared error when trying to build a wasm module with nottinygo. I have imported nottinygo in main.go and used tinygo build tags as per the README. nottinygo version is 0.7.1

I was use this library with a wasi module written in Go to enhance the performance, but i encounter an error with
panic: runtime error: divide by zero
when i use compress/gzip stdlib to decode gzip bytes.

the full compile command:
tinygo build -o wasm.wasm -target wasi -gc=custom -tags=custommalloc -target=wasi -scheduler=none ./main.go

i run it with wasmtime:
env WASMTIME_BACKTRACE_DETAILS=1 wasmtime wasm.wasm

panic: runtime error: divide by zero
Error: failed to run main module `wasm.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x24d61 - runtime.abort
                           at /root/software/tinygo/src/runtime/runtime_tinygowasm.go:70:6              - runtime.runtimePanicAt
                           at /root/software/tinygo/src/runtime/panic.go:71:7
           1: 0x27519 - runtime.divideByZeroPanic
                           at /root/software/tinygo/src/runtime/panic.go:177:16
           2: 0x1cf57 - runtime.alloc
                           at /root/workplace/go_poject/GOPATH/pkg/mod/github.com/wasilibs/[email protected]/intmap.go:93:19
           3: 0x23408 - compress/flate.newHuffmanEncoder
                           at /root/.g/go/.versions/1.21.3/src/compress/flate/huffman_code.go:60:24
           4: 0x22609 - compress/flate.generateFixedOffsetEncoding
                           at /root/.g/go/.versions/1.21.3/src/compress/flate/huffman_code.go:95:24              - runtime.run
                           at /root/software/tinygo/src/runtime/scheduler_none.go:24:9              - _start
                           at /root/software/tinygo/src/runtime/runtime_wasm_wasi.go:21:5

when i remove the gzip lib, everything is working fine

tinygo version:
tinygo version 0.30.0 linux/amd64 (using go version go1.21.3 and LLVM version 16.0.1)

Should this work on browser?

I have tried to use nottinygc in our WASI project, but it fails.

With the standard tinygo GC the execution runs successfully.

Tested using @wasmer/wasi npm package and also https://github.com/bjorn3/browser_wasi_shim

We are using tinygo 0.28 (current dev branch).

Any hint is welcome! Thanks.

TinyGo Version: 0.28.1
nottinygc Version: v0.4.0

when cgo LDFLAGS is
#cgo LDFLAGS: -Lwasm -lgc -lmimalloc -lclang_rt.builtins-wasm32 --export=malloc --export=free
the wasm plugin will cause memory leak.

but when I remove --export=malloc --export=free it will not.

What't the reason?

Hello,
I was trying out this library with a WASM module written in Go and compiled with the -scheduler=asyncify TinyGo build option.

When running the WASM module with wasmtime the module exits with:

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0xa142 - <unknown>!GC_expand_hp_inner
           1: 0x10db3 - <unknown>!GC_init
           2: 0x1220b - <unknown>!GC_generic_malloc_inner
           3: 0x127b4 - <unknown>!GC_generic_malloc
           4: 0x12924 - <unknown>!GC_malloc
           5: 0x31050 - runtime.alloc
                           at ~/go/pkg/mod/github.com/wasilibs/[email protected]/gc.go:53:6
           6: 0x53096 - <unknown>!runtime.run$1
           7: 0x52c5a - <goroutine wrapper>
                           at /usr/local/Cellar/tinygo/0.29.0/src/runtime/scheduler_any.go:23:2
           8: 0x12b84 - <unknown>!tinygo_launch
           9: 0x52b10 - (*internal/task.Task).Resume
                           at /usr/local/Cellar/tinygo/0.29.0/src/internal/task/task_asyncify.go:109:17              - runtime.scheduler
                           at /usr/local/Cellar/tinygo/0.29.0/src/runtime/scheduler.go:236:11              - runtime.run
                           at /usr/local/Cellar/tinygo/0.29.0/src/runtime/scheduler_any.go:28:11              - _start
                           at /usr/local/Cellar/tinygo/0.29.0/src/runtime/runtime_wasm_wasi.go:21:5
    2: wasm trap: integer divide by zero

TinyGo Version: 0.29.0 darwin/amd64 (using go version go1.20.1 and LLVM version 15.0.0)
nottinygc Version: v0.4.0

Any advice is much appreciated.

Hi @anuraaga !

I am running into a strange behaviour when my wasm plugin executes a DispatchHttpCall. It appears the call to the service (dispatch call) is never made in majority of the runs, making me think if the gc clears the context before the execution of the dispatch call.

The flow works as expected when i don't run the custom gc
Req -> parseHeaders -> dispatchCall(ServiceA) -> callback -> getResponseHeaders -> update headers -> ReqGoesUpstream

The metrics below measure the execution_count(execution of the plugin) & the success_count(completion of callback). Below are the numbers when I don't run the custom gc - execution_count = success_count.

Whereas when i run the custom gc, only a portion of requests end up making a call upstream (I verified that requests don't reach serviceA)

My main.go has these imports

import (
	"crypto/rand"
	"fmt"
	"math/big"
	"os"
	"strings"
	"time"

	"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
	"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
	"github.com/tidwall/gjson"
	_ "github.com/wasilibs/nottinygc"
	"golang.org/x/exp/slices"
)

And the file ends with this after all the functions

// required for nottinygc
//
//export sched_yield
func sched_yield() int32 {
	return 0
}

Appreciate your thoughts on this behaviour , and thank you for this package :)

Description

I am developing a WASM plugin for Istio ingress gateway. In the WASM Plugin, the fastcache is used for caching and the total cache size is limit to 32MB for preventing the OOM, however, the memory usage still keeps increasing

How to reproduce

dependecy:

• tinygo v0.28.1
• nottinygc v0.4.0
• proxy-wasm-go-sdk v0.22.0
• istion v1.18.0

The code of plugin to describe the issuse :

proxy-wasm-memory-leak

I`ve modify the source code of fastcache to make it can be compiled with tinygo

The Test script:

func TestMemoryLeak(t *testing.T) {
// Ingress Host and port :192.168.58.3:31505
	req, err := http.NewRequest("GET", "http://192.168.58.3:31505", nil)
	if err != nil {
		panic(err)
	}
	for i := 0; i < 10000000; i++ {
		_, err := http.DefaultClient.Do(req)
		if err != nil {
			continue
		}
	}
	fmt.Println("done")
}

what I’ve inspect is that is memory is keep increasing and never release

HI @anuraaga, thank you for releasing such a great GC strategy in proxy-wasm!

I am having trouble with the default GC in TinyGo(v0.27.0) as well.
When I try to compile my wasm plugin use nottinygc and follow the instructions in the readme, the compilation works. But when I deploy it into a k8s cluster with Istio(v1.14 and v1.16), the envoy shows the error when loading the wasm module.

Here is the error message inside envoy proxy.

Failed to load Wasm module due to a missing import: wasi_snapshot_preview1.sched_yield

Do you have any insights for such an issue? Appreciate your help!

Hi @anuraaga, thanks for your project, we are using this nottinygc in our higress project which based on Envoy's wasm extension mechanism.

We discovered a memory leak during use, and you can find demo code to reproduce the memory leak here:
https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions/gc-test

Note here that we replace nottinygc with github.com/higress-group/nottinygc, if we comment out this replacement like this:

...
// replace github.com/wasilibs/nottinygc v0.5.1 => github.com/higress-group/nottinygc v0.0.0-20231019105920-c4d985d443e1

require (
	github.com/alibaba/higress/plugins/wasm-go v0.0.0
	github.com/tetratelabs/proxy-wasm-go-sdk v0.22.0
	github.com/tidwall/gjson v1.14.3
)
...

Then build the wasm file and start envoy with the following configuration:

node:
  cluster: test-cluster
  id: test-idn

admin:
  address:
    socket_address: { address: 127.0.0.1, port_value: 9901 }

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 127.0.0.1, port_value: 10000 }
    per_connection_buffer_limit_bytes: 1024000000
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          http_filters:
          - name: gctest
            typed_config:
              "@type": type.googleapis.com/udpa.type.v1.TypedStruct
              type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
              value:
                config:
                  name: basic_auth
                  vm_config:
                    runtime: envoy.wasm.runtime.v8
                    code:
                      local:
                        filename: ./mem.wasm
                    allow_precompiled: true
                  fail_open: true
                  configuration:
                    "@type": type.googleapis.com/google.protobuf.StringValue
                    value: '{"bytes": 10485760}'
          - name: envoy.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains:
              - "*"
              routes:
              - match:
                  prefix: /
                direct_response:
                  status: 200
                  body:
                    inline_string: "hello world"

Use the following command to perform a stress test:

wrk -R 50 -t 10 -c 10 -d60s -s post.lua http://localhost:10000/

After a few seconds, we can see the following log:

[2023-10-23 15:42:30.858][1902490][info][wasm] [source/extensions/common/wasm/context.cc:1204] wasm log basic_auth: [gc-test] {"Sys": 1063452672,"HeapSys": 1058013184,"HeapIdle": 145313792,"HeapInuse": 0,"HeapReleased": 0}
[2023-10-23 15:42:30.861][1902490][error][wasm] [source/extensions/common/wasm/wasm_vm.cc:41] Function: proxy_on_request_headers failed: Uncaught RuntimeError: unreachable\nProxy-Wasm plugin in-VM backtrace:\n  0:  0x1d4fc - runtime._panic\n  1:  0x1ce84 - runtime.alloc\n  2:  0x26758 - main.onHttpRequestHeaders\n  3:  0x3761a - proxy_on_request_headers

Hi, when I use v0.4.0, I meet this error

github.com/wasilibs/[email protected]/init.go:9:14: invalid flag: --export=malloc

How can I fix it?

When I implemented the envoy proxy-wasm plugin, I found that if the response body is processed by gzip compression, a memory leak will occur. This problem also exists in the coraza-waf plugin. I suspect it may be affected by non-ASCII characters. You can use these test server program to reproduce it:

ASCII Response:

package main

import (
        "fmt"
        "log"
        "math/rand"
        "net/http"
        "time"
)

func echoHandler(w http.ResponseWriter, r *http.Request) {
        maxSize := 100 * 1024              // 100KB
        randSize := rand.Intn(maxSize) + 1 

        characters := "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 ,.!?;:"

        rand.Seed(time.Now().UnixNano())

        randomText := make([]byte, randSize)
        for i := range randomText {
                randomText[i] = characters[rand.Intn(len(characters))]
        }

        w.Header().Set("Content-Type", "text/plain")

        _, err := w.Write(randomText)
        if err != nil {
                fmt.Println("Error writing body")
                return
        }
}

func main() {
        http.HandleFunc("/", echoHandler)

        fmt.Println("Server is running on port 5000...")

        log.Fatal(http.ListenAndServe(":5000", nil))
}

Non ASCII Response

package main

import (
        "fmt"
        "log"
        "math/rand"
        "net/http"
)

func echoHandler(w http.ResponseWriter, r *http.Request) {
        maxSize := 100 * 1024              // 100KB
        randSize := rand.Intn(maxSize) + 1 
        randomData := make([]byte, randSize)
        _, err := rand.Read(randomData) 
        if err != nil {
                http.Error(w, "Failed to generate random data", http.StatusInternalServerError)
                return
        }

        _, err = w.Write(randomData)
        if err != nil {
                fmt.Println("Error writing body")
                return
        }

}
func main() {
        http.HandleFunc("/", echoHandler)
        fmt.Println("Server is running on port 5000...")
        log.Fatal(http.ListenAndServe(":5000", nil))
}

I am using malloc with TinyGo but with nottinygc it panics.

How can I resolve this?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1199875]

`// These function are exported by TinyGo
malloc := mod.ExportedFunction("malloc")
free := mod.ExportedFunction("free")

// Allocate Memory
results, err := malloc.Call(ctx, nameSize)
if err != nil {
	log.Panicln(err)
}
namePosition := results[0]

// This pointer is managed by TinyGo,
// but TinyGo is unaware of external usage.
// So, we have to free it when finished
defer free.Call(ctx, namePosition)

`