bind_tools NotImplementedError when using ChatOllama about langchain

[tcztzy]

You can use Ollama's OpenAI compatible API like

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    api_key="ollama",
    model="llama3",
    base_url="http://localhost:11434/v1",
)
llm = llm.bind_tools(tools)

Pretend the Ollama mode as OpenAI and have fun with the LLM developping!

[lalanikarim]

#20881 (merged) already added bind_tools feature into OllamaFunctions
#21625 (pending merge) adds support for tool_calls

[lalanikarim]

@Vishnullm
After reviewing the underlying the code for Agent and AgentExecutor, I've concluded that more work is needed on OllamaFunctions, such as stream support, before it can be used within AgentExecutor.
Until that work is done, unfortunately, you won't be able to use AgentExecutor based agents.
LangGraph based agents do work however, and you should be able to modify existing LangGraph agents or build new ones that use OllamaFunctions.

[KadriMufti]

Are there any updates in this space? I am also using Ollama and I have tried the various suggestions here, including the first one of using ChatOpenAI, and I get errors. Thank you for your efforts @lalanikarim

[RyanKung]

I finally solved this issue by using litellm

ollama -> litellm proxy -> langchain's ChatOpenAI

[msssouza]

I finally solved this issue by using litellm

ollama -> litellm proxy -> langchain's ChatOpenAI

Hi. Can you share your code?

[sbusso]

@hyhzl, no random mention, please.

[subhash137]

Any one gt the solution for that

[subhash137]

Even structuredoutut is not working

error -

[subhash137]

Thank you for your reply

…

On Sat, 11 May 2024, 13:12 Tang Ziya, ***@***.***> wrote: You can use Ollama's OpenAI compatible API <https://ollama.com/blog/openai-compatibility> like from langchain_openai import ChatOpenAIllm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", )llm = llm.bind_tools(tools) Pretend the Ollama mode as OpenAI and have fun with the LLM developping! — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHNN3T6STIR6I6LHJSTZBXDVBAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGYYTQMRTG4> . You are receiving this because you commented.Message ID: ***@***.***>

[alexanderp99]

@subhash137 . According to the Ollama docs, their Chat Completions API does not support function calling yet. Did you have any success?

[subhash137]

Yes, I did.

…

On Mon, 13 May 2024, 19:24 Alexander, ***@***.***> wrote: @subhash137 <https://github.com/subhash137> . According to the Ollama docs <https://github.com/ollama/ollama/blob/main/docs/openai.md>, their Chat Completions API does not support function calling yet. Did you have any success? — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHLMHTGSQSIIM7XMBDDZCDAYTAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBXGY2DCNZXGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[alexanderp99]

@subhash137 would you please show, how you achieved function calling in that way?

[kaminwong]

@subhash137 would you please show, how you achieved function calling in that way?

tcztzy's comment should work

[subhash137]

from langchain_openai import ChatOpenAI llm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", ) llm = llm.bind_tools(tools) If you want to run locally use LM studio download the models , run the server give the api to base_url. but i prefer to use groq for faster and efficient output.

…

On Tue, 14 May 2024 at 13:17, KM ***@***.***> wrote: @subhash137 <https://github.com/subhash137> would you please show, how you achieved function calling in that way? tcztzy's comment should work — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHNVDN2RXAOE2HL4Z3LZCG6PTAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZGUYDMNBQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kaminwong]

@subhash137 would you please show, how you achieved function calling in that way?

Oh sorry I just tried, seems the tools are not invoked this way. Did someone successfully make the model use the tools provided?

[subhash137]

Oh 😳 , I am sorry I didn't implemented it correctly. Just now I ran the code it is running successfully but tools are not invoked. Aa.. I don't have any idea now

…

On Tue, 14 May 2024, 16:21 KM, ***@***.***> wrote: @subhash137 <https://github.com/subhash137> would you please show, how you achieved function calling in that way? Oh sorry I just tried, seems the tools are not invoked this way. Did someone successfully make the model use the tools provided? — Reply to this email directly, view it on GitHub <#21479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AW6YNHNRKDUWHP6VCW45263ZCHUCZAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZHA4DQMRRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AmirMohamadBabaee]

I faced this error too. is there any quick fix for this problem? using OllamaFunctions can fix it?

[Harsh-Kesharwani]

@lalanikarim can i use chat model along with function calling. As i see chatOllama does not supports bind_tools, but in documentation it is given how to bind_tools with chatOllama.

[ErfanMomeniii]

We should use OllamaFunctions and pass the LLaMA model name as a parameter, as it includes a suitable bind_tools method for adding tools to the chain. The ChatOllama class does not possess any methods for this purpose.
for more details see https://python.langchain.com/v0.1/docs/integrations/chat/ollama_functions
Alternatively, we can manage this manually by defining new classes that inherit from ChatOllama, incorporating tools as parameters, and creating an appropriate invoke function to utilize these tools.
@Harsh-Kesharwani

[lalanikarim]

@lalanikarim can i use chat model along with function calling. As i see chatOllama does not supports bind_tools, but in documentation it is given how to bind_tools with chatOllama.

@Harsh-Kesharwani
Like @ErfanMomeniii suggested, you can use OllamaFunctions if you need function calling capabilities with Ollama.
OllamaFunctions inherits from ChatOllama and adds newer bind_tools and with_structured_output functions as well as adds tool_calls property to AIMessage.
While you can currently already use OllamaFunctions for function calling, there is an unmerged PR #21625 that fixes the issue where you want a chat response from OllamaFunctions in case none of the provided functions are appropriate for the request. I am hoping that to be merged sometime this week.

[ntelo007]

Can someone please post a mini example of tool calling with these pr merges?

[KIC]

I am still not able to get it to work:

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool


@tool
def magic_function(input: int):
    """applies magic function to an input"""
    return input * -2

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad")
    ]
)

tools = [magic_function]

model = OllamaFunctions(
    model="llama3",
    # formal="json",    # commented or not, does not change the error
    keep_alive=-1,
    temperature=0,
    max_new_tokes=512,
)

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input":"What is the value of magic_function(3)"})

TypeError: Object of type StructuredTool is not JSON serializable

[lalanikarim]

I am still not able to get it to work:

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool


@tool
def magic_function(input: int):
    """applies magic function to an input"""
    return input * -2

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad")
    ]
)

tools = [magic_function]

model = OllamaFunctions(
    model="llama3",
    # formal="json",    # commented or not, does not change the error
    keep_alive=-1,
    temperature=0,
    max_new_tokes=512,
)

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input":"What is the value of magic_function(3)"})

TypeError: Object of type StructuredTool is not JSON serializable

This PR fixed the JSON serialization error and couple other things.

#22339

[lalanikarim]

Example notebook with tool calling from withing LangGraph agent.
https://github.com/lalanikarim/notebooks/blob/main/LangGraph-MessageGraph-OllamaFunctions.ipynb

Since #22339 is not yet merged, the notebook installs langchain-expermental from my repo (source for #22339).

[Harsh-Kesharwani]

@lalanikarim does agent carries context that is return by tool at every iteration, suppose i have 3 tools below is the execution flow:

agent...
use tool 1
tool 1 response: resp1

agent... (does agent carries resp1 or summarization or knowledge graph for it)
use tool 2
tool 2 response: resp2

agent... (does agent carries resp1, resp2 or summarization or knowledge graph for it)
use tool 3
tool 3 response: resp3

The question is does agent carries tool response as a context for next iteration.

[lalanikarim]

@lalanikarim does agent carries context that is return by tool at every iteration, suppose i have 3 tools below is the execution flow:

agent...
use tool 1
tool 1 response: resp1

agent... (does agent carries resp1 or summarization or knowledge graph for it)
use tool 2
tool 2 response: resp2

agent... (does agent carries resp1, resp2 or summarization or knowledge graph for it)
use tool 3
tool 3 response: resp3

The question is does agent carries tool response as a context for next iteration.

@Harsh-Kesharwani

You provide an initial state on every irritation. Unless you pass the previous context into the next iteration, the agent starts with a fresh state every time. I hope this answers your question.

initial_state = ...
updated_state = agent.invoke(initial_state)

next_initial_state = <combine updated_state and a new initial state>
updated_state = agent.invoke(next_initial_state)

[Harsh-Kesharwani]

@lalanikarim can i log the prompt which is passed to the agent.

[lalanikarim]

@lalanikarim can i log the prompt which is passed to the agent.

@Harsh-Kesharwani
I have included langtrace links for multiple runs in the notebook. Take a look and let me know if that answers your questions.

[lalanikarim]

Take a look at #22339 which should have addressed this issue.
The PR was approved and merged yesterday but a release is yet to be cut from it and should happen in the next few days.

In the meantime, you may try and install langchain-experimental directly from langchain's source like this:

pip install git+https://github.com/langchain-ai/langchain.git\#egg=langchain-experimental\&subdirectory=libs/experimental

I hope this helps.

[Vishnullm]

@lalanikarim

I have been following your work to make tool calling happen for the OLLAMA.

https://github.com/langchain-ai/langchain/blob/master/libs/experimental/langchain_experimental/llms/ollama_functions.py

I have used the sorce code mentioned in the link for the OllamaFunctions. Eventhough I am able to convert the tools, I am still getting the error "Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable
". The mistake maybe from my side but I am unable to figure it out. Below is the error for my code:

**Successfully converted tool: {'name': 'sql_db_query', 'parameters': {'title': '_QuerySQLDataBaseToolInput', 'type': 'object', 'properties': {'query': {'title': 'Query', 'description': 'A detailed and correct SQL query.', 'type': 'string'}}, 'required': ['query']}, 'description': "Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields."}

Successfully bound tools to LLM
Successfully created agent
Successfully created agent executor

Entering new AgentExecutor chain...
Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable**

My code:
from langchain_experimental.llms.ollama_functions import OllamaFunctions, convert_to_ollama_tool
from langchain.agents import Tool, create_tool_calling_agent, AgentExecutor

llm = OllamaFunctions(model = "llama3", format = "json", temperature = 0, keep_alive=-1)
toolkit = SQLDatabaseToolkit(db=db, llm=llm, use_query_checker=True)
tools = toolkit.get_tools()

converted_tools = []
for tool in tools:
try:
converted_tool = convert_to_ollama_tool(tool)
print(f"Successfully converted tool: {converted_tool}")
converted_tools.append(converted_tool)
except Exception as e:
print(f"Error converting tool: {tool}, Error: {e}")

try:
llm_with_tools = llm.bind_tools(converted_tools)
print("Successfully bound tools to LLM")
except Exception as e:
print(f"Error binding tools to LLM: {e}")

SQL_PREFIX = """You are an agent designed to interact with a SQL database.
Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer.
Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results.
You can order the results by a relevant column to return the most interesting examples in the database.
Never query for all the columns from a specific table, only ask for the relevant columns given the question.
You have access to tools for interacting with the database.
Only use the below tools. Only use the information returned by the below tools to construct your final answer.
You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.

DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.

To start you should ALWAYS look at the tables in the database to see what you can query.
Do NOT skip this step.
Then you should query the schema of the most relevant tables.
You have access to the following tools:"""

prompt = ChatPromptTemplate.from_messages(
[
("system", SQL_PREFIX),
("human", "{input}"),
("placeholder", "{agent_scratchpad}"),
]
)

try:
agent = create_tool_calling_agent(llm=llm_with_tools, tools=tools, prompt=prompt)
print("Successfully created agent")
except Exception as e:
print(f"Error creating agent: {e}")

print("Tools passed to AgentExecutor:", converted_tools)

try:
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
print("Successfully created agent executor")
except Exception as e:
print(f"Error creating agent executor: {e}")

question = "give me only the attachment attempts for the left teats where the cow id with highest value "

try:
response = agent_executor.invoke({"input": question})
print("Successfully executed agent")
print(response)
except Exception as e:
print(f"Error executing agent: {e}")

[lalanikarim]

@lalanikarim

I have been following your work to make tool calling happen for the OLLAMA.

https://github.com/langchain-ai/langchain/blob/master/libs/experimental/langchain_experimental/llms/ollama_functions.py

I have used the sorce code mentioned in the link for the OllamaFunctions. Eventhough I am able to convert the tools, I am still getting the error "Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable
". The mistake maybe from my side but I am unable to figure it out. Below is the error for my code:

**Successfully converted tool: {'name': 'sql_db_query', 'parameters': {'title': '_QuerySQLDataBaseToolInput', 'type': 'object', 'properties': {'query': {'title': 'Query', 'description': 'A detailed and correct SQL query.', 'type': 'string'}}, 'required': ['query']}, 'description': "Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields."}

Successfully bound tools to LLM
Successfully created agent
Successfully created agent executor

Entering new AgentExecutor chain...
Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable**

My code:
from langchain_experimental.llms.ollama_functions import OllamaFunctions, convert_to_ollama_tool
from langchain.agents import Tool, create_tool_calling_agent, AgentExecutor

llm = OllamaFunctions(model = "llama3", format = "json", temperature = 0, keep_alive=-1)
toolkit = SQLDatabaseToolkit(db=db, llm=llm, use_query_checker=True)
tools = toolkit.get_tools()

converted_tools = []
for tool in tools:
try:
converted_tool = convert_to_ollama_tool(tool)
print(f"Successfully converted tool: {converted_tool}")
converted_tools.append(converted_tool)
except Exception as e:
print(f"Error converting tool: {tool}, Error: {e}")

try:
llm_with_tools = llm.bind_tools(converted_tools)
print("Successfully bound tools to LLM")
except Exception as e:
print(f"Error binding tools to LLM: {e}")

SQL_PREFIX = """You are an agent designed to interact with a SQL database.
Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer.
Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results.
You can order the results by a relevant column to return the most interesting examples in the database.
Never query for all the columns from a specific table, only ask for the relevant columns given the question.
You have access to tools for interacting with the database.
Only use the below tools. Only use the information returned by the below tools to construct your final answer.
You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.

DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.

To start you should ALWAYS look at the tables in the database to see what you can query.
Do NOT skip this step.
Then you should query the schema of the most relevant tables.
You have access to the following tools:"""

prompt = ChatPromptTemplate.from_messages(
[
("system", SQL_PREFIX),
("human", "{input}"),
("placeholder", "{agent_scratchpad}"),
]
)

try:
agent = create_tool_calling_agent(llm=llm_with_tools, tools=tools, prompt=prompt)
print("Successfully created agent")
except Exception as e:
print(f"Error creating agent: {e}")

print("Tools passed to AgentExecutor:", converted_tools)

try:
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
print("Successfully created agent executor")
except Exception as e:
print(f"Error creating agent executor: {e}")

question = "give me only the attachment attempts for the left teats where the cow id with highest value "

try:
response = agent_executor.invoke({"input": question})
print("Successfully executed agent")
print(response)
except Exception as e:
print(f"Error executing agent: {e}")

@Vishnullm
I'll try this later today and provide an update.

[lalanikarim]

@Vishnullm
The issue is because create_tool_calling_agent rebinds the tools to the llm and doesn't give you the opportunity to convert_to_ollama_tool.

I am investigating a more permanent fix, but for now, make the following adjustments to your code:

Original

agent = create_tool_calling_agent(llm=llm_with_tools, tools=tools, prompt=prompt)

New

from langchain.agents.format_scratchpad.openai_tools import (
    format_to_openai_tool_messages,
)
from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(
            x["intermediate_steps"]
        ),
    }
    | prompt
    | llm_with_tools
    | OpenAIToolsAgentOutputParser()
)

This code is from the custom codes notebook

[Vishnullm]

@lalanikarim , Thanks for the suggestion.
My requirement is to use mainly llama llms from Ollama. With the suggestion you provided I can only use gpt models. Is there any other way around?

I tried create_sql_agent with zeroshotpromptreactdescription. When I give any complex question which requires multiple sql tables to join or to look for multiple columuns, the agent just fails and goes in a loop of redoing the same action execution even though it sometime generate the answer it does not know that it is the correct answer.

As a result, I thought create_tool_calling_agent would be more effective for this use case, but I am facing the issue with function calling when I used OllamFunctions.

llm = OllamaFunctions(model = "llama3", format = "json", temperature = 0, keep_alive=-1)
toolkit = SQLDatabaseToolkit(db=db, llm=llm, use_query_checker=True)
tools = toolkit.get_tools()

converted_tools = [convert_to_ollama_tool(tool) for tool in tools]
llm_with_tools = llm.bind_tools(tools=converted_tools)
agent = create_tool_calling_agent(
    llm=llm_with_tools,
    tools=converted_tools,  
    prompt=prompt
)

agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)

when I use this code by understanding your work on OllamaFunctions wrapper, i am getting the error:
Ollama call failed with status code 400. Details: {"error":"invalid options: functions"}

I believe with this structure atleast .bind_tools() is working but not the tool_calling.

Please guide me on how to make a model suitable for sql Q/A using Ollama or local llms.

[lalanikarim]

@Vishnullm
I am looking into getting OllamaFunctions working correctly with tool calling. This might take some time.
I have created a notebook with a simple zero shot LangGraph agent that you might want to check out in the mean time.
https://github.com/lalanikarim/notebooks/blob/main/SQL-LangGraph-Agent/SqlAgent-LangGraph.ipynb

It can currently answer simple questions but can be adapted to answer more complex questions by applying techniques like ReAct etc.

I hope this helps.

[Vishnullm]

Hi @lalanikarim,

my use case is to specifically use open source llms for interacting with yaml files which have the same format of key value pairs across all the files with constant key names. so i thought if i convert all these files to sql tables then can have more functionality for the chatbot that interacts with sql database. when I created the create_sql_agent with zeroshotreactdesription agent type, the model is not working really well posing me with errors as shuch shown in the below picture

As I am using llama3 which I see as a good open source llm so far, I am not able to use tool calling or open ai function agent types which are termed as best agent types online.

1. as per your understanding, can you tell which agent and agenttype is best suitable for sql chatbots and y they are best?
2. If you have any process in mind that is best suitable for the sql chat model, please share your thoughts.

your expert guidance is really appreciable. thanks for the support.

[lalanikarim]

Hi @lalanikarim,

my use case is to specifically use open source llms for interacting with yaml files which have the same format of key value pairs across all the files with constant key names. so i thought if i convert all these files to sql tables then can have more functionality for the chatbot that interacts with sql database. when I created the create_sql_agent with zeroshotreactdesription agent type, the model is not working really well posing me with errors as shuch shown in the below picture

As I am using llama3 which I see as a good open source llm so far, I am not able to use tool calling or open ai function agent types which are termed as best agent types online.

1. as per your understanding, can you tell which agent and agenttype is best suitable for sql chatbots and y they are best?

2. If you have any process in mind that is best suitable for the sql chat model, please share your thoughts.

your expert guidance is really appreciable. thanks for the support.

@Vishnullm @KadriMufti
I have not yet tried local llms for SQL generation within agents. Having said that, couple of things I do want to mention:

1. The AgentExecutor based agents are being deprecated in favor of LangGraph (https://python.langchain.com/v0.2/docs/how_to/agent_executor/)
2. You may want to try llms fine tuned for coding tasks, like codestral etc as they may perform better on SQL tasks.
3. You may also want to investigate llms fine tuned for SQL tasks for your use case.

OllamaFunctions only supports LangGraph agents. AgentExecutor based agents are not yet supported. And given that LangChain team is deprecating them, I suggest looking into SQL agents built with LangGraph.

Here is yet to be published documentation for OllamaFunctions which also includes instructions for building LangGraph agents.
https://langchain-git-fork-lalanikarim-ollama-function-fbb983-langchain.vercel.app/v0.2/docs/integrations/chat/ollama_functions/

I hope it helps.