Serving with custom engine about clearml-serving

[Bunoviske]

Hello Martin!

No particular reason, I just wanted to understand how flexibel ClearML Serving is. Regarding your second question, I think I already got good outcome from our slack conversation and here: #32

Once you have the new example with custom engine, let me know! Thanks :)

[bmartinn]

Quick update @Bunoviske
You can now find a custom serving example here 😃

[bmartinn]

Hi @Bunoviske

I checked the pipeline example where you use custom engine

Yes it is in he to do list to add a specific custom model serving example. I'll update here once it is up

What if I want to run normal pytorch inference without any engine?

So you can just do "pytorch inference" as a custom engine (basically import pytorch and run the model. That said Triton is so much faster than just using vanilla pytorch for inference, and this is basically out of the box. Any reason for manually running the model inference ?

Customizable RestAPI for serving (i.e. allow per model pre/post-processing for easy integration). How can a really customize the RestAPI?

The idea is the fact you can have any python code as pre/post processing, which really customize the entire API. For example, RestAPI gets a url as string, and the preprocessing would download it, return a numpy object, then the inference is done on the image.
What exactly are you aiming for with the customization?