Native executable quarkus-amazon-lambda images built on Mac M1 Pro not working on AWS about quarkus

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @gastaldi (m1), @matejvasek (amazon-lambda,funqy), @patriot1burke (amazon-lambda,funqy), @zakkak (mandrel,native-image)

[zakkak]

@bredlej judging by the error message:

Error: fork/exec /var/task/bootstrap: exec format error

I suspect that what's happening is that you are building a native image targeting your Mac's AArch64 architecture and then you try to run it on AWS on an x86 machine.

Can you please run the following on your Mac?:

file <path-to-the-generated-binary>

And also report what AWS instance you are using for running the lambdas?

[bredlej]

@zakkak Thanks for responding!

You're right, the image is the following:
target/aws-lambda-test-1.0.0-SNAPSHOT-runner: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=d6277eaab2d4c70f1e5b1cec62615cb5bf169538, not stripped

In the weekend I read about doing multistage docker builds, so I managed to do a workaround by doing the following:

1. Create a Dockerfile.multistage file as described at https://quarkus.io/guides/building-native-image#multistage-docker
2. Run docker -D build -f src/main/docker/Dockerfile.multistage -t aws/aws-lambda-test-multistage .
3. Create repository on AWS aws ecr create-repository --repository-name <repository-name>
4. Tag the docker build: docker tag <docker build sha> <aws-account-nr>.dkr.ecr.<aws-region>.amazonaws.com/<repository-name>:v1
5. Login to AWS: aws ecr get-login-password --region <aws-region> | docker login --username AWS --password-stdin <aws-account-nr>.dkr.ecr.<aws-region>.amazonaws.com
6. docker push <aws-account-nr>.dkr.ecr.<aws-region>.amazonaws.com/<repository-name>:v1
7. Go to AWS Management Console / Lambda / Functions -> Create function...
8. Chose to create from Container Image and architecture arm64

Today I'm trying to get it working from the CLI by adjusting the ./target/manage.shscript, but I'm getting nowhere yet.
Basically I tried to build a native image with: quarkus build --native --no-tests -Dquarkus.native.container-build=true
and edting the ./target/manage.sh by adding --architectures arm64 to the aws lambda create-function call.
I had to comment out RUNTIME=provided from line 73 because it said it's not compatible with arm64 for some reason.

But still no success:

> ./target/manage.sh native invoke

Invoking function
++ aws lambda invoke response.txt --cli-binary-format raw-in-base64-out --function-name AwsLambdaTestNative --payload file://payload.json --log-type Tail --query LogResult --output text
++ base64 --decode
START RequestId: f13655f3-45c8-423f-9e55-99012cf5a129 Version: $LATEST
Class not found: io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler: java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler
	at java.base/java.net.URLClassLoader.findClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Unknown Source)
	at java.base/java.lang.Class.forName(Unknown Source)

END RequestId: f13655f3-45c8-423f-9e55-99012cf5a129
REPORT RequestId: f13655f3-45c8-423f-9e55-99012cf5a129	Duration: 583.06 ms	Billed Duration: 584 ms	Memory Size: 256 MB	Max Memory Used: 95 MB	Init Duration: 264.63 ms
{"errorMessage":"Class not found: io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler","errorType":"java.lang.ClassNotFoundException"}%

So while the multistage docker build approach works, I wonder it's still possible somehow to get it done via the manage.sh script.

Do you have perhaps any ideas what to try out?

EDIT:

To add to this - the same build works if not done natively:

1. quarkus build
2. ./target/manage.sh create
3. ./target/manage.sh invoke

Result:

Invoking function
++ aws lambda invoke response.txt --cli-binary-format raw-in-base64-out --function-name AwsLambdaTest --payload file://payload.json --log-type Tail --query LogResult --output text
++ base64 --decode
__  ____  __  _____   ___  __ ____  ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2024-02-12 12:45:48,977 INFO  [io.quarkus] (main) aws-lambda-test 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.7.2) started in 1.105s.
2024-02-12 12:45:48,978 INFO  [io.quarkus] (main) Profile prod activated.
2024-02-12 12:45:48,979 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, cdi]
START RequestId: 0ac72d5f-ba0a-4604-868e-0f4179eae053 Version: $LATEST
END RequestId: 0ac72d5f-ba0a-4604-868e-0f4179eae053
REPORT RequestId: 0ac72d5f-ba0a-4604-868e-0f4179eae053	Duration: 129.53 ms	Billed Duration: 130 ms	Memory Size: 256 MB	Max Memory Used: 129 MB	Init Duration: 1460.61 ms
"Hello Bill"%

So there's something going on when doing a native build.

[zakkak]

Do you have perhaps any ideas what to try out?

Unfortunately I am not familiar with AWS lambdas and the referenced scripts, so someone else will need to step in. Perhaps @matejvasek or @patriot1burke

[deki]

Did you try to add

        Architectures:
          - arm64

to sam.native.yaml (see
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-resource-function-properties for more details)?

java17 has been made default in the meantime so this is no longer an issue...

[Karm]

Hello, @bredlej,
There is very little magic going on. Quarkus just creates a zip file with your executable and conf and that zip file is then uploaded to AWS and AWS just runs the executable on a Linux system. That's it. So it matters what Linux system is the executable built for, both CPU architecture and GLibc version. I encountered this kind of issue when I was testing #23998

You need to configure your AWS Lambda to e.g. use Linux 1 or Linux 2, depending on what you target. You also need it to configure correctly either amd64 (intel) or aarch64 (ARM).

Your local Mac OS container build most likely uses Linux aarch64 builder image, so your binary would be Linux aarch64 one.

I am closing this as a non-Quarkus issue.