Make it easier to use local LLMs by decoupling AutoGPT from dependence on OpenAI function calling about autogpt

[joshuacox]

I wanted add in the @Wladastic 's comment. I like the idea of using multiple models, perhaps eventually a model could be trained specifically for the functions themselves.

[joshuacox]

@ntindle can you give a link to the discord discussion here for those of use arriving from the web at large? Or maybe a summarication here? I think this is a very important topic going forward, this project should not be shackled to openAI.

[ntindle]

Absolutely. Good point. I'll try to be more diligent about that in the future too.

[ntindle]

Link to start of discussion: https://discord.com/channels/1092243196446249134/1095817829405704305/1212845507060437033

Summary:

Do you need function calls

Yes, we need function calls. No, it doesn't have to be natively supported in the model. They can look at the OpenAIProvider with _functions_compat_fix_kwargs and _tool_calls_compat_extract_calls for a very rough idea of how it could be done.

I think the "middleware" idea does make sense.

fwiw yesterday I did manage to get the generate_agent_profile_for_task call working with a locally-running llava-v1.5-7b-q4 model after some hacking. Gonna try to see how far I get with making a full session work today and then I'd be happy to brainstorm about how to make it easier for others to do this with their own local models via middleware or more docs or both.

However one question is, is it worth investing refactoring time in the AutoGPT bot itself or is this functionality that should actually just land in the Forge part?

However one question is, is it worth investing refactoring time in the AutoGPT bot itself or is this functionality that should actually just land in the Forge part?

It makes sense to put the work into AutoGPT right now. We are planning on moving library code including autogpt.core.resource.model_providers to Forge, in the near future. The cleaner the module, the easier the move.

@Pwuts then put this on the roadmap as #7001

[Wladastic]

I just went and ran it myself again, I can confirm the LLM doesnt even respond in the correct format.
What does work though is the one-shot prompt. I think for using local llm's we could at least use these as a first?
It just works for a few steps only though as anything above 3000 tokens is too confusing for Mistral already

[ntindle]

responded on discord :)

[Wladastic]

I wanted add in the @Wladastic 's comment. I like the idea of using multiple models, perhaps eventually a model could be trained specifically for the functions themselves.

Thank you :)
I am already working on another ai project due to that.
I figured out a very good way to avoid the down sides. Multi step works amazingly well with neural chat 3.1, capybara mistral7b, gemma 2b and mixtral 8x7b. Funnily gguf models work best.
It requires a shit ton of optimization but its doable. You just have to figure out the params needed for each call

[joshuacox]

@Wladastic are you working out of a branch? Or are you able to get all this working on main? And/Or do you have a link to this other project?

[Wladastic]

@joshuacox
I wrote a smaller version, working on autogpt directly was too complicated and painful for that haha

[Wladastic]

@joshuacox
I wrote a smaller version, working on autogpt directly was too complicated and painful for that haha

[joshuacox]

@Wladastic I completely understand, sometimes you need to simplify things to isolate the parts you are working with. I encourage you to put up a branch or repo, it might be easier for some of us to contribute to as well.

[Wladastic]

@Wladastic I completely understand, sometimes you need to simplify things to isolate the parts you are working with. I encourage you to put up a branch or repo, it might be easier for some of us to contribute to as well.

I could try to but my project is now merged with my own ai that works different than AutoGPT right now.
I can try to make a simplified version.

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.