ChatHuggingFace + HuggingFaceEndpoint does not properly implement `max_new_tokens` about langchain

[keenborder786]

Okay, you are comparing two different things. The Huggingface Inference Client returns the following object, which has an attribute of usage that is of type ChatCompletionOutputUsage.

The ChatCompletionOutputUsage has three types of token usage:

1. completion_tokens: This is the number of tokens required to complete the prompt. In your case, this is always fixed because you are calling the same prompt to complete. Try something else, and it should change.
2. prompt_tokens: The number of tokens in the prompt.
3. total_tokens: The sum of completion_tokens and prompt_tokens.

So, you are implicitly comparing the total_tokens through the n_tokens function with completion_tokens, which is incorrect. You should compare the total_tokens attribute to make the correct comparison.

P.S. I double-checked the LangChain code and ensured that ChatHuggingFace is returning the correct ChatCompletionOutputUsage without any modification.

[BobMerkus]

Okay, you are comparing two different things. The Huggingface Inference Client returns the following object, which has an attribute of usage that is of type ChatCompletionOutputUsage.

The ChatCompletionOutputUsage has three types of token usage:

1. completion_tokens: This is the number of tokens required to complete the prompt. In your case, this is always fixed because you are calling the same prompt to complete. Try something else, and it should change.
2. prompt_tokens: The number of tokens in the prompt.
3. total_tokens: The sum of completion_tokens and prompt_tokens.

So, you are implicitly comparing the total_tokens through the n_tokens function with completion_tokens, which is incorrect. You should compare the total_tokens attribute to make the correct comparison.

P.S. I double-checked the LangChain code and ensured that ChatHuggingFace is returning the correct ChatCompletionOutputUsage without any modification.

I think you are misunderstanding the example code, the n_tokens() function is called on the output of the chain contents, thus the completion_tokens == n_tokens(output) - 1. The -1 one is the special end of sequence token (this is why the output says 101 tokens, not 100). The problem is that the ChatCompletionOutputUsage.output_tokens should always be less or equal to max_new_tokens, but this is 100 tokens regardless of the supplied max_new_tokens.

[TiagoPinaC]

I'm having the same problem... Did you find a solution?

[BobMerkus]

I'm having the same problem... Did you find a solution?

No I have not, this issue renders the entire Huggingface x Langchain implementation obsolete to me. I have been attempting to work around the issue using an OpenAI compatible web server through either LlamaCpp/Ollama.