jataware / archytas Goto Github PK
View Code? Open in Web Editor NEWGiving LLM agents access to tools so they perform open ended tasks
License: GNU General Public License v3.0
Giving LLM agents access to tools so they perform open ended tasks
License: GNU General Public License v3.0
There needs to be some way for the LLM to lookup documentation for a particular tool at runtime rather than including all the information in the initial prompt. When there are lots of tools, or the tools have complex behavior with long prompts, this takes up lots of the limited input context available in GPT-4 (currently 8k tokens).
The current approach of stuffing everything into the initial prompt has the following drawbacks:
PythonTool
or similar? Want to be able to pass in local variables, that the model could presumably lookup the values of. This is semi-handled by the pyman tool: https://github.com/jataware/archytas/blob/master/archytas/tools.py#L134, though the prompt generated for the PythonTool
doesn't update to indicate that the locals exist, or that information about them can be looked upAs the react loop "thoughts" are being shared, the agent could summarize the previous tools output as part of its though and return the summary as part of the thoughts, that would really help with transparency, without worrying about cluttering up the human chat with too many details.
When a tool
is defined inside of an agent, it seems to be called incorrectly.
The specific case I ran into:
Error while handling message llm_request in function llm_request
jupyter-1 | Traceback (most recent call last):
jupyter-1 | File "/home/jupyter/.local/lib/python3.10/site-packages/archytas/react.py", line 204, in react_async
jupyter-1 | tool_fn = self.tools[tool_name]
jupyter-1 | KeyError: 'search_installed_packages'
jupyter-1 |
jupyter-1 | During handling of the above exception, another exception occurred:
jupyter-1 |
jupyter-1 | Traceback (most recent call last):
jupyter-1 | File "/home/jupyter/.local/lib/python3.10/site-packages/beaker_kernel/lib/utils.py", line 30, in wrapper
jupyter-1 | result = await fn(self, message)
jupyter-1 | File "/home/jupyter/.local/lib/python3.10/site-packages/beaker_kernel/kernel.py", line 440, in llm_request
jupyter-1 | result = await self.context.agent.react_async(request, react_context={"message": message})
jupyter-1 | File "/home/jupyter/.local/lib/python3.10/site-packages/archytas/react.py", line 206, in react_async
jupyter-1 | action_str = await self.error(f'Unknown tool "{tool_name}"\nAvailable tools: {", ".join(self.tools.keys())}')
jupyter-1 | File "/home/jupyter/.local/lib/python3.10/site-packages/archytas/react.py", line 302, in error
jupyter-1 | raise FailedTaskError(f"Too many errors during task. Last error: {mesg}")
jupyter-1 | archytas.react.FailedTaskError: Too many errors during task. Last error: Unknown tool "search_installed_packages"
jupyter-1 | Available tools: Agent.ask_user, Agent.get_available_functions, Agent.get_functions_docstring, Agent.retrieve_documentation_for_module, Agent.search_installed_packages, Agent.submit_code
This agent class has a tool defined inside of it like this
@tool()
def search_installed_packages(self, code:str) -> str:
....
and it shows up in the list of available functions. However, the agent is unable to call the tool in the action loop
Presently, the prompts generated for an agent are static, and based solely off the docstrings/type signatures of the tools passed into the ReActAgent class. E.g.
from archytas.tools import PythonTool
from archytas.react import ReActAgent
agent = ReActAgent(tools=[PythonTool], verbose=True)
print(agent.prompt)
Which generates the following (abridged) prompt:
You are the ReAct (Reason & Action) assistant. ...
# Tools
You have access to the following tools which can help you in your job:
PythonTool (class):
Tool for running python code. If the user asks you to write code, you can run it here.
methods:
run:
Runs python code in a python environment.
The environment is persistent between runs, so any variables created will be available in subsequent runs.
The only visible effects of this tool are from output to stdout/stderr. If you want to view a result, you MUST print it.
_input_: (str) The code to run
_output_: (str) The stdout output of the code
ask_user:
Ask the user a question and get their response.
You should ask the user a question if you do not have enough information to complete the task, and there is no suitable tool to help you.
_input_: (str) The question to ask the user
_output_: (str) The user's response
final_answer:
the final_answer tool is used to indicate that you have completed the task. You should use this tool to communicate the final answer to the user.
_input_: the final answer to the user's task
fail_task
the fail_task tool is used to indicate that you have failed to complete the task. You should use this tool to communicate the reason for the failure to the user. Do not call this tool unless you have given a good effort to complete the task.
_input_: the reason for the failure
<other prompt information...>
This prompt gets generated when the agent is instantiated, and there is no way to adjust it after the fact. For the PythonTool
specifically, it would be useful to be able modify the description to indicate that prelude code was run or that extra local variables are available.
# make a tool instance with prelude code and variables
python = PythonTool(
# code to run at the beginning
prelude='import numpy as np\nfrom matplotlib import pyplot as plt',
# variables to include in the environment
locals={'fib_n': fib_n, 'ModelSimulation': ModelSimulation, 'jackpot': jackpot, 'pyman': pyman},
)
agent = ReActAgent(tools=[python], verbose=True)
At present, there is no way to update the description for the PythonTool to reflect that the environment has extra variables, and has already run some code. It's also relevant to managing context for a given tool.
There needs to be a good approach for adjusting the tool's prompt
@tool
/@toolset
wrappers that get generated, and attach a _prompt_override
variable to the wrapper.
_prompt_override
variable is present, it is used instead of the generated prompt@toolsets
could allow for an optional generate_prompt
method that would be called when prompts are being generated, and override the default generated prompt for the toolset.gpt-4 turbo outputs json markdown given current prompt. We can either filter the response or use the official open ai format enforcing -
completion = openai.chat.completions.create(
**model='gpt-4-1106-preview',**
response_format={"type": "json_object"},
messages=[
{
"role": "user",
"content": "How do I output all files in a directory using Python? Answer in json",
},
],
)
completion.choices[0].message.content
Better to keep consistent with majority of projects and dev expectations. Using master
gives this repo an antiquated/outdated feel.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.