Newer Base Image KServe Container fails with exec /usr/local/bin/dockerd-entrypoint.sh: exec format error about serve

[agunapal]

Thanks for reporting..looking into this. Able to repro the error. Earlier we didn't move to 22.04 as the ubuntu 22.04 runners were flaky. I will try running CI on 22.04 to see if its resolved now.

[tylertitsworth]

Before it gets asked here, yes I have tried to capture logs from within the deployed container, however the container does not even start so no other logs are recorded (other than the liveness probe and queue-proxy failing and all of that)

[agunapal]

@tylertitsworth Please pull the submodules before you build kfs image

git submodule update --init --recursive

I am able to build it with 22.04 after doing this

 docker image inspect pytorch/torchserve-kfs:latest-cpu | grep "org.opencontainers.image.version"
                "org.opencontainers.image.version": "22.04"

[tylertitsworth]

@agunapal In the build script I use to build this container I pull submodules (https://github.com/intel/ai-containers/blob/main/pytorch/serving/build-kfs.sh#L9)

I am able to build the container, however, my issue is when it is deployed to k8s.

[tylertitsworth]

@agunapal any update on this? Is there any misunderstanding I can help alleviate?

[agunapal]

Hi @tylertitsworth I understand the problem. I will get back to you this week.

[agunapal]

On ubuntu 22.04, tried running grpc testcases..these worked

test_gRPC_inference_api.py::test_inference_apis PASSED                                                                                                                             [ 21%]
test_gRPC_inference_api.py::test_inference_stream_apis 2024-04-06T18:20:11,945 [INFO ] W-9024-echo_stream_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9024-echo_stream_1.0-stderr
PASSED                                                                                                                      [ 21%]
test_gRPC_inference_api.py::test_inference_stream2_apis PASSED                                                                                                                     [ 22%]
test_gRPC_management_apis.py::test_management_apis PASSED        

So, it may be something specific to docker/kserve.. Will try the steps you have mentioned