[BUG] Update Training and Serving Merlin on AWS SageMaker using latest merlin image about merlin

[edknv]

I'm working on updating the merlin-tensorflow image to 23.06 here: #1040.

After bumping the image version to 23.06 and updating the processing workflow in train.py to reflect recent changes, and running the updated example on AWS, we are getting an error:

Failed to transform operator <merlin.systems.dag.runtimes.triton.ops.workflow.TransformWorkflowTriton object at 0x7fe7df82a160>

RuntimeError: Failed for execute the inference request. Model '0_transformworkflowtriton' is not ready.

which doesn't tell us much what is going wrong. I'll try to run the container locally to debug.

[rnyak]

I'm working on updating the merlin-tensorflow image to 23.06 here: #1040.

After bumping the image version to 23.06 and updating the processing workflow in train.py to reflect recent changes, and running the updated example on AWS, we are getting an error:

Failed to transform operator <merlin.systems.dag.runtimes.triton.ops.workflow.TransformWorkflowTriton object at 0x7fe7df82a160>

RuntimeError: Failed for execute the inference request. Model '0_transformworkflowtriton' is not ready.

which doesn't tell us much what is going wrong. I'll try to run the container locally to debug.

thanks @edknv !

[wei-m-teh]

@edknv are there any update on this issue? I am trying to deploy a Merlin model to Sagemaker following the example given. I am running into the same issue.

[edknv]

@wei-m-teh Apologies for the delay. It's in review at the moment, but I updated #1040 with a workaround I found for making the notebook work with the latest 23.08 image.

[rnyak]

@wei-m-teh can you please test this PR at your end? thanks.