chatglm output results are repeating with basic prompts about openvino.genai

[peterchen-intel]

It may due to benchmark.py force to output "--infer_count" tokens for performance consistency (with fixed output size instead of stop at end_token). Will add an option to stop_ending_token.

[avinashbhat09]

Tried the same OV converted model using chatglm-openvino (https://github.com/OpenVINO-dev-contest/chatglm3.openvino) and it works fine. We dont see any repetitive words.

With this we can conclude below:

1. No issue running inference on CPU or GPU

2. Not a model issue

3. No quantization issue

This looks more of gen-ai interfacing with the model.

[avinashbhat09]

Can we add chatbot kind of implementation same as chatglm-openvino into the gen-ai to support chatglm?

[avinashbhat09]

Hi, any update on this? @Wovchena

[Wovchena]

Hi. I don't have any update. @peterchen-intel is the correct person to discuss llm_bench related questions with. As for the chatbot kind of implementation, the sample is here https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/chat_sample.

[avinashbhat09]

Thanks @Wovchena . Unfortunately, the chat sample does not work for me.

[avinashbhat09]

Hi. I don't have any update. @peterchen-intel is the correct person to discuss llm_bench related questions with. As for the chatbot kind of implementation, the sample is here https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/chat_sample.

@peterchen-intel : Any input from your side?

[Wovchena]

Your model is stateless. You need a stateful one. To export such a model, ensure you don't have --disable-stateful while running optimum-cli export openvino. Alternatively, if you use python ./llm_bench/python/convert.py, you need to specify --stateful (and not --disable-stateful).

[avinashbhat09]

Thanks for your input @Wovchena . I converted using this command and it worked.
optimum-cli export openvino --trust-remote-code --model THUDM/chatglm3-6b chatglm3-6b_stateful --task question-answering

So this confirms that we dont have issues with model or quantization. Coming back to original bug- when we use benchmark.py why do we see the answers repeating? Anything can be done to fix that? For our validation get the metrices printed for each response is important (like token/sec, first token latency etc). Which is currently not available in chat_sample.

[peterchen-intel]

CVS-146307

[peterchen-intel]

#606

[peterchen-intel]

@avinashbhat09 Can you try HEAD of openvino.genai master branch with option --end_token_stopping?