Feature request: Support fp16 with self-speculative decoding on XPU in ipex_llm.serving.fastchat.ipex_llm_worker about bigdl

[gc-fu]

Hi, this issue actually contains two parts:

1. A bug that is caused by using low_bit fp16 in ipex_llm_worker.
2. Feature request: support ipex_llm_worker with speculative decoding.

The first part have been fixed by this pr #10907.

The second part will be supported by @hzjane.

[brosenfi]

Just to provide a bit more information @gc-fu - here, it's providing the torch_dtype value as "auto", but in the fp16 example with self-speculative decoding, it's showing that torch_dtype should be set to torch.float16. Also there are other parameters in the example that aren't being provided when launching via the ipex_llm_worker - specifically "speculative" and "optimize_model" - this is why I marked this as a feature request and not a bug, I thought this mode just isn't supported yet for the ipex_llm_worker module (would be nice if it was though).

[hzjane]

@brosenfi
The self-speculative decoding using fastchat worker will be supported in this PR.
But the speculative example only supports running on intel max GPU due to the memory usage limitations. You can try it on max GPU or CPU later.

[brosenfi]

Thank you @gc-fu

[gc-fu]

Hi, I am working on to reproduce this issue.