Settings menu for custom model names about koalaclient

[jackschedel]

I'll leave the issue open, since I think it is a good change. (I have alot of issues open that I'll never spend the time to implement but maybe someone else will take pity on me and help xD)

Please let me know if you need any help navigating the codebase, I know learning React can be pretty daunting for beginners.

[jackschedel]

btw, i'm working on this myself now, so no need to try. development is occurring on the 2.0.9 branch

[jackschedel]

Thanks for your input. Btw, 2.1.0 is still in development, not in a working state yet.

[jackschedel]

@imesha10
Hi, I don't ever use custom proxies, but I'd be willing to accept a PR for this implemented as a settings menu for it like there is with Prompt Library (ideally with the ability to hide certain default ones in the dropdown.) with the ability to specify token pricing and max tokens.

From a code standpoint, this will require refactoring away from the custom ModelChoice type, probably just to a standard string.

[imesha10]

@imesha10 Hi, I don't ever use custom proxies, but I'd be willing to accept a PR for this implemented as a settings menu for it like there is with Prompt Library (ideally with the ability to hide certain default ones in the dropdown.)

From a code standpoint, this will require refactoring away from the custom ModelChoice type, probably just to a standard string.

Alright, I'll try to do a pr in the comming weeks. I have very little experiece with node and related but I will be able to figure it out.

Please close this issue when you see it and I will reopen a pull request once I'm done.

If I don't then assume that I failed.

(oh and when I meant proxies, what I meant was custom openai endpoints.)

Thanks!

[imesha10]

btw, i'm working on this myself now, so no need to try. development is occurring on the 2.0.9 branch

Ok that's good. Thank you.

[imesha10]

@jackschedel
Oh ya by the way if you didn't know. OpenAI default endpoint and most openai proxies gives a list of all the available models via the /v1/models endpoint.
So all you have to do is send a call to that and it will return a json object that you can then parse into the application.
This is how applications like SillyTavern and other ones work.

You can find more info in here https://platform.openai.com/docs/api-reference/models

If you want to see examples in a different language then press this drop box and choose:

[jackschedel]

I'm instead implementing the ability to do fully custom model names, so KoalaClient can be used with any endpoint.

[imesha10]

I'm instead implementing the ability to do fully custom model names, so KoalaClient can be used with any endpoint.

Ya it's definitely a very useful feature to be able to enter custom model names so it can be used with any endpoint type and it basically will fix my problem of being unable to use certain models.

However I should mention that after looking at this project (KoalaClient) and using it for a while I have some thoughts.
I think you should definitely after this feature or some features down the line consider making it so that it auto populates the model list for the ones that has that feature.

I'm taking a blind guess but I'm guessing most of your users will be openai (85%) anthropic/claude(10%), and other(5%).

So if you implement this later you would be able to get most of your users all the models they have access to (all openai models from /v1/models). This makes it so that most of your users will not have to go into the api connection menu and manually input their specific models which will make the app feel simple and very easy to use.

Currently KoalaClient can't seem to fetch other openai models and it's limited to:

Here is a similar one where it auto fetches in SillyTavern via a proxy:

I just complied the latest version 3.0.9 and it does not seem to have the model "gpt-4-0125-preivew". So if you were to auto populate the list you would not have to constantly update it with the latest and the best models at least for openai.

Anyways these are just thoughts and I might be wrong or thinking in the wrong direction. Thanks for you time.