Fix stop list of string in VLLM generate about langchain

[keenborder786]

okay I am checking it

[keenborder786]

Can you please share your code and prompt as well and the output which shows how the stop word is not working?

[keenborder786]

Because I double checked the code, everything seems to be functional.

[NikitaKlichko]

Because I double checked the code, everything seems to be functional.

When specifying stops in the config and calling the list of stop tokens does not work, because in this case the stop list will be None in generate params (see code above in description issue)

generation_config = {
    'temperature':0.01,
    'top_p':0.9,
    'top_k':30,
    'max_tokens':1024,
    'repetition_penalty':1.1,
    'stop': ['.']
    }
    llm = VLLM(model=llm_name, dtype='float16', **generation_config)
    llm.invoke('question')

if we do that

llm.invoke('question', stop=['...'])

it works.
For the same reason, if we run, for example, rag_chain stop list don't work

rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

[keenborder786]

No it won't be None because when you pass stop to the instance of VLLM i.e during the initialization, the self.stop is passed to the VLLM client by using the _default_params attribute. This is passed here, so it cannot be None regardless of whether you pass the stop during initialization or during invoking.

[NikitaKlichko]

No it won't be None because when you pass stop to the instance of VLLM i.e during the initialization, the self.stop is passed to the VLLM client by using the _default_params attribute. This is passed here, so it cannot be None regardless of whether you pass the stop during initialization or during invoking.

if do like this stop token won't work in current _generate implementation

generation_config = {
    'temperature':0.01,
    'top_p':0.9,
    'top_k':30,
    'max_tokens':1024,
    'repetition_penalty':1.1,
    'stop': ['.']
    }
 llm = VLLM(model=llm_name, dtype='float16', **generation_config)
    
SYSTEM = "You are an astronomer scientist. Explain step by step"
question = "How does the Moon differ from Earth?"
template = [{"role": "system", "content": f"{SYSTEM}"}, 
            {"role": "user", "content": "Вопрос: {}".format(question)}]

llm_tokenizer = AutoTokenizer.from_pretrained(llm_name)
prompt = llm_tokenizer.apply_chat_template(template, tokenize=False, add_generation_prompt=True)
llm.invoke(prompt)
---------------------
ANSWER:
The Moon and Earth are two celestial bodies that are closely related yet distinct in many ways. Here's a step-by-step explanation of the main differences between them:

1. **Origin**: The Moon is thought to have formed about 4.5 billion years ago, shortly after the formation of the Earth. One theory is that the Moon was created when a massive object collided with the Earth, causing debris to be thrown into orbit and eventually coalesce into the Moon.

2. **Size and Mass**: The Moon is significantly smaller than Earth. It has a diameter of about 3,474 kilometers (2,159 miles), which is about one-quarter of Earth's diameter. The Moon's mass is about one-eighteenth of Earth's mass.
 and so on.

Check #15921

"""In the line params = {**self._default_params, **kwargs, "stop": stop}, the "stop" parameter from the "_default_params" method is overwritten by the local "stop" parameter. If no "stop" parameter is passed to the "_generate" method, it defaults to None, effectively ignoring the "stop" parameter set in the VLLM class"""