how to use hf client about dspy

[okhat]

You’re right about this. We are pushing a number of fixes for HF server and clients soon. We’re just starting to play with them — expect many rough edges for a couple of weeks, but then should become something stable

[ColinS97]

Update: I just figured out the reason it is not working was somehow related to my usage of ngrok, if I have the the server and client running on the same machine it works fine. I will try to get it running through ngrok as well now.

[okhat]

Thanks for looking into this! This HF side of DSP is not very mature right now, so let us know if you face any issues. The FLAN models may not be ideal here based on our initial testing

[ColinS97]

Why are the FLAN models not ideal? Just because they lack the "emergent abilities" to really make use of DSP programs or some technical reason? I wasn't planning on using FLAN for my experiments but Llama and Open Assistant type models. So I am wondering if there is anything i should watch out for.

[okhat]

Yeah, you may see better results with decoder-only models, like the ones you mentioned, especially if you use closed LLMs as the demonstrator. We’ll share more soon

[ColinS97]

Very nice, I am excited to see you further progress.
(I hope "compiling" LoRa adapters on consumer GPUs is somewhere on your roadmap, since that would be a really exciting feature for me)

In my fork of DSP I have now adapted the hf_client.py a little bit so that you can pass an ngrok url to the object and it will just work with the hf_server.py.

This is very useful for me since I am serving the models in a HPC cluster an then ngrok them out to the public internet so I can access my models from any device so I think this may be useful for others as well :)

[okhat]

Yup we already have these LoRAs — @paridhimaheshwari2708 maybe we should merge something soon?

[ColinS97]

I noticed that when using hf_client that my kwargs are not sent to the hf_server. This is a problem when trying the self-consistency use cases. Thus in hf_client in the generate function I changed:
From:
payload = {"prompt": prompt, **kwargs}
To:
payload = {"prompt": prompt, "kwargs": kwargs}

This seemed to fix the problem.
Can you tell me the reason why the ** operator was there initially?

[okhat]

By the way things are much more stable now with TGI client in DSPy. Feel free to let us know if you face problems in the new version of the framework.