compiler(arm64) overflow in call relocations about wazero

[davidmdm]

I've fixed this issue for myself by switching to using the interpreter which may in fact be more appropriate for my use-case overall since startup time is more important to me than execution time.

However will leave this up, and will answer any questions. I want to mention that I love wazero and think it is an understated contribution to the Go-Ecosystem. Well done team!

[evacchi]

so just to give you some feedback, I've been looking into this. The bad news is that I am still figuring it out, the somewhat good news is that this bug is limited to arm64. With shorter examples; e.g.

	c := v1.ContainerPort()
	fmt.Printf("%v", c)

I can get a stack trace similar to the one you show:

error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
	.github.com_google_gnostic_models_openapiv2.__Any_.ProtoReflect(i32) i32
	.runtime.doInit1(i32) i32
	.runtime.main(i32) i32
	.wasm_pc_f_loop()
	._rt0_wasm_wasip1()

in general it looks like something funky is going on in the init() phase.

This won't happen either with the interpreter or GOARCH=amd64

[evacchi]

@davidmdm in case you didn't see this yet, I have a WIP PR addressing your issue #2169

[evacchi]

Hi, can you share the faulty module so we can inspect it?
EDIT: silly me I didn't see the link in the top post, thanks. How was the binary produced?

[evacchi]

• for reference, just running through wazero run pg.wasm shows the same behavior (no need for the wrapping script).
• Also, GOARCH=amd64 wazero run pg.wasm crashes just the same, which is likely to mean a bug in the front-end (i.e. it is not a bug in the machine-specific codegen) EDIT 2 days later: LOL GOARCH=amd64 wazero is a no-op 🤦 this is still running on arm64. in fact, the bug is definitely machine-specific as amd64 is not affected
• finally, some experimental "fix" that I am working on for a different bug in the front-end (re: bound check elimination) causes the hard crash to turn into a wasm stack trace; it might not be related and just be a bug on top of another bug 😬

error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
	.k8s.io_api_apps_v1beta2.__DeploymentList_.DeepCopyInto(i32) i32
	.runtime.doInit1(i32) i32
	.runtime.main(i32) i32
	.wasm_pc_f_loop()
	._rt0_wasm_wasip1()

EDIT: it looks like, before this commit eb345e3, we had a different issue:

error instantiating wasm binary: module[] function[_start] failed: wasm error: unreachable
wasm stack trace:
	.k8s.io_api_authentication_v1.__TokenReviewStatus_.Unmarshal(i32) i32
	.runtime.doInit1(i32) i32
	.runtime.main(i32) i32
	.wasm_pc_f_loop()
	._rt0_wasm_wasip1()

@davidmdm I think you should be able to reduce the reproducer to a more manageable size: it looks to me like it is entirely related to (un)marshalling the huge k8s protos. Can you try, e.g. marshall/unmarshall a fixed JSON payload and see if that blows everything up again?

thanks!

[davidmdm]

@evacchi

The binary was produced via the Go Toolchain:

GOOS=wasip1 GOARCH=wasm go build -o pg.wasm ./cmd/pg

I tried running wazero directly against an equivalent pg.wasm and got the same thing you did:

wazero run pg.wasm 
error instantiating wasm binary: module[] function[_start] failed: wasm error: out of bounds memory access
wasm stack trace:
        .k8s.io_api_apps_v1beta2.__DeploymentList_.DeepCopyInto(i32) i32
        .runtime.doInit1(i32) i32
        .runtime.main(i32) i32
        .wasm_pc_f_loop()
        ._rt0_wasm_wasip1()

I then tried it on a "more manageable" hello-world version of a kubernetes resource but still compiling the kubernerte's client-go packages:

package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"os"

	appsv1 "k8s.io/client-go/applyconfigurations/apps/v1"
	corev1 "k8s.io/client-go/applyconfigurations/core/v1"
	metav1 "k8s.io/client-go/applyconfigurations/meta/v1"
)

func main() {
	if err := run(); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

func run() error {
	name := os.Args[0] + "-sample-app"
	labels := map[string]string{"app": name}

	replicas := flag.Int("replicas", 2, "desired replica count for deployment")

	flag.Parse()

	dep := appsv1.Deployment(name, "").
		WithLabels(labels).
		WithSpec(
			appsv1.DeploymentSpec().
				WithReplicas(int32(*replicas)).
				WithSelector(metav1.LabelSelector().WithMatchLabels(labels)).
				WithTemplate(
					corev1.PodTemplateSpec().
						WithLabels(labels).
						WithSpec(
							corev1.PodSpec().WithContainers(
								corev1.Container().
									WithName(name).
									WithImage("alpine:latest").
									WithCommand("watch", "echo", "hello", "world"),
							)),
				),
		)

	return json.NewEncoder(os.Stdout).Encode(dep)
}

and although it didn't crash with a stack trace, no output is rendered with v1.7.0, but it works successfully with v1.6.0 using wazero run on the wasm outputted via GOOS=wasip1 GOARCH=wasm go build

[evacchi]

hah, I thought pg had something to do with postgres, but even this smaller example produces a 62meg binary :D so it really has to do with instantiating these huge protos. See e.g.

package main

import (
	corev1 "k8s.io/client-go/applyconfigurations/core/v1"
)

func main() {
	c := corev1.Container().
		WithName("hello").
		WithImage("alpine:latest").
		WithCommand("watch", "echo", "hello", "world")

	println(c)
}

produces:

unexpected fault address 0x1408a156004
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x1408a156004 pc=0x152774fa4]
...

[davidmdm]

Pg does have something to do with postgres, but regardless of what I am deploying/generating, I am generating kubernetes manifests from K8 client libraries via the magic of wazero! I've switched to using the interpreter for now since its a good fit for my use-case, but it's good you've managed to find a more minimalistic way to reproduce the issue!