salesforceairesearch / agentlite Goto Github PK

View Code? Open in Web Editor NEW

376.0 376.0 34.0 6.45 MB

License: Apache License 2.0

Python 6.48% Jupyter Notebook 93.52%

agentlite's People

Contributors

Stargazers

Watchers

agentlite's Issues

Local Model Inference for webshop

In benchmark/evaluate_webshop, you link lines of code where supposedly we only need to change the base_url and model_name. But, this won't work unless the model is defined in agentlite.llm.agent_llms.

For instance, if I go to AgentLite/benchmark/webshop and run

python evaluate_webshop.py --llm xlam --lam_url http://localhost:8000/v1

I get the error openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': 'Only llama&&mixtral allowed now, your model xlam', 'code': 40301}

Can the documentation to this framework please be released?

Finish action customization.

If I want to an agent to output multiline code, it is hard to use current framework .

that is because the syntax:

    Finish [ {"response": "content" } ]

the code should be included in the content, for gpt-3.5 , multiline content inside a json formatted string is error-prone.

If ,the customization code can easily :

re-parse the llm-output if a action parser fails
specify other format other than current FinishAction

it will be easier.

运行Quick Start命令，执行卡住

如图，执行了“python3 ./example/SearchManager.py”后,会直接卡住不动，求问是什么原因。
环境：
MacBook Pro Apple M2 Pro
Python 3.11.0

[Contribution Request] Integrate Unify AI into AgentLite

Hello SalesforceAIResearch team,

I'm Kato Steven Mubiru, an AI engineer contributing to Unify AI. I'm reaching out to explore the possibility of integrating Unify into AgentLite

Before proceeding with a detailed proposal, I'd like to inquire about:

Your current process for accepting external contributions, especially for significant integrations.
Any specific guidelines or preferences for proposing and implementing new features.
The best way to discuss the potential benefits and implementation details of this integration.
Our goal would be to enhance AgentLite with Unify's capabilities.

https://unify.ai

I'd be happy to provide more details about Unify and our integration ideas once I understand your preferred process for such contributions.

Thank you for your time and guidance.

Best regards,
Kato Steven Mubiru
Unify AI

DuckDuckGo RateLimit Exception on running example/SearchManager.py

While running the SearchManager example, rate limit exception arises despite of running it the very first time as well. I think the Agent should be programmed to handle this exception as well. Please let me know if you could fix this? Thank you very much

I would be happy to fix it as well and create a PR if you think this could be helpful. Here is the error trace for your reference

python example/SearchManager.py
Agent Search_Manager receives the following TaskPackage:
[
        Task ID: c07e4633-5147-481e-8a64-8bbc736300b4
        Instruction: what is micorsoft famous for
]
====Search_Manager starts execution on TaskPackage c07e4633-5147-481e-8a64-8bbc736300b4
====                                                                                   /opt/conda/envs/agentlite/lib/python3.11/site-packages/langchain_core/_api/deprecation.
py:117: LangChainDeprecationWarning: The function `run` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.                                     warn_deprecated(
Agent Search_Manager takes 0-step Action:
{
        name: Think
        params: {'response': "I should first ask duck_search_agent to search for inform
ation about Microsoft, if I don't find an answer, I will ask wiki_search_agent to search for information about Microsoft"}                                                    }
Observation: OK
Agent Search_Manager takes 1-step Action:
{
        name: duck_search_agent
        params: {'Task': 'search the information about Microsoft'}
}
Agent duck_search_agent receives the following TaskPackage:
[
        Task ID: c07e4633-5147-481e-8a64-8bbc736300b4
        Instruction: search the information about Microsoft
]
====duck_search_agent starts execution on TaskPackage c07e4633-5147-481e-8a64-8bbc73630
0b4====                                                                                Agent duck_search_agent takes 0-step Action:
{
        name: DuckDuckGo_Search
        params: {'query': 'Microsoft information'}
}
Traceback (most recent call last):
  File "/home/ubuntu/AgentLite/example/SearchManager.py", line 82, in <module>
    test_manager_agent()
  File "/home/ubuntu/AgentLite/example/SearchManager.py", line 77, in test_manager_agen
t                                                                                          response = search_manager(test_task_pack)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/AgentLite/agentlite/agents/BaseAgent.py", line 115, in __call__
    self.execute(task)
  File "/home/ubuntu/AgentLite/agentlite/agents/BaseAgent.py", line 150, in execute
    observation = self.forward(task, action)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/AgentLite/agentlite/agents/ManagerAgent.py", line 157, in forward
    observation = agent(new_task_package)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/AgentLite/agentlite/agents/BaseAgent.py", line 115, in __call__
    self.execute(task)
  File "/home/ubuntu/AgentLite/agentlite/agents/BaseAgent.py", line 150, in execute
    observation = self.forward(task, action)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/AgentLite/agentlite/agents/BaseAgent.py", line 238, in forward
    observation = action(**agent_act.params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/AgentLite/example/SearchActions.py", line 20, in __call__
    return self.search.run(query)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/langchain_core/tools.py"
, line 401, in run                                                                         raise e
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/langchain_core/tools.py"
, line 358, in run                                                                         self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/langchain_community/tool
s/ddg_search/tool.py", line 64, in _run                                                    res = self.api_wrapper.results(query, self.max_results, source=self.backend)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/langchain_community/util
ities/duckduckgo_search.py", line 111, in results                                          for r in self._ddgs_text(query, max_results=max_results)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/langchain_community/util
ities/duckduckgo_search.py", line 48, in _ddgs_text                                        ddgs_gen = ddgs.text(
               ^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search.py", line 57, in text                                                          return self._run_async_in_thread(super().text(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search.py", line 53, in _run_async_in_thread                                          result = future.result()
             ^^^^^^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/concurrent/futures/_base.py", line 456
, in result                                                                                return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/agentlite/lib/python3.11/concurrent/futures/_base.py", line 401
, in __get_result                                                                          raise self._exception
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search_async.py", line 167, in text                                                   results = await self._text_api(keywords, region, safesearch, timelimit, max_results
)                                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^                                                                                        File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search_async.py", line 258, in _text_api                                              raise e
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search_async.py", line 253, in _text_api                                              await asyncio.gather(*tasks)
  File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search_async.py", line 229, in _text_api_page                                         resp_content = await self._aget_url("GET", "https://links.duckduckgo.com/d.js", par
ams=payload)                                                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^                                                                             File "/opt/conda/envs/agentlite/lib/python3.11/site-packages/duckduckgo_search/duckdu
ckgo_search_async.py", line 124, in _aget_url                                              raise RatelimitException(f"{resp.url} {resp.status_code} Ratelimit")
duckduckgo_search.exceptions.RatelimitException: https://links.duckduckgo.com/d.js?q=Mi
crosoft+information&kl=wt-wt&l=wt-wt&p=&s=0&df=y&vqd=4-7297422369529734716423891034146014912&ex=-1 202 Ratelimit

Suggestion: Replace DuckSearchAgent with WikiSearchAgent in example/SearchAgent.py

Everytime I pull the recent updates from Agentlite master some or the other problem happens in example/SearchAgent.py because of DuckSearchAgent. I think it is better to make WikiSearchAgent as the default Agent as it is much more stable. Please let me know your thoughts on the same

编码问题

[Execution]
Task:请为新来的员工小明办理入职手续，并发放发本月的工资
Action:人事专员[{"Task": "\u4e3a\u5c0f\u660e\u529e\u7406\u5165\u804c\u624b\u7eed"}]
Observation: 入职手续已经办理完成。

Action:

log打印的日志，如果是中文的话可能出现编码异常，包括给出的哲学家辩论那个例子，prompt的有关信息改成中文后就会出现编码异常，可能会影响模型效果

Running evaluate_hotpot_qa.py throws an error

Hey guys,
I appreciated reading your paper. However I just wanted to see your eval results for myself and running the script results in the follwoing error when I execute python evaluate_hotpot_qa.py --llm gpt-4o --agent_arch act. I have implemented an adapter
for AzureOpenAI in order to use gpt-4o which I have tested with your framework beforehand and worked well.

Connected to pydev debugger (build 232.9559.58)
Processing:   0%|          | 0/100 [00:00<?, ?it/s]Agent wiki_search_agent receives the following TaskPackage:
[
	Task ID: 03244001-d72a-4b5e-b5a1-bf22bde7636d
	Instruction: Which magazine was started first Arthur's Magazine or First for Women?
]
====wiki_search_agent starts execution on TaskPackage 03244001-d72a-4b5e-b5a1-bf22bde7636d====
Agent wiki_search_agent takes 0-step Action:
{
	name: Think
	params: {'response': "I need to search Arthur's Magazine and First for Women, find their start dates, and then compare them to determine which was started first."}
}
Observation: OK
Agent wiki_search_agent takes 1-step Action:
{
	name: Wikipedia_Search
	params: {'query': "Arthur's Magazine"}
}
Observation: Arthur's Magazine (1844–1846) was an American literary periodical published in Philadelphia in the 19th century. Edited by Timothy Shay Arthur, it featured work by Edgar A. Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas G. Spear, and others. In May 1846 it was merged into Godey's Lady's Book.
A few years later Arthur would launch a new publication entitled Arthur's Home Magazine.
Agent wiki_search_agent takes 2-step Action:
{
	name: Wikipedia_Search
	params: {'query': 'First for Women'}
}
Observation: First for Women is a woman's magazine published by A360media in the US. The magazine was started in 1989 by Bauer Media Group. In 2011 the circulation of the magazine was 1,310,696 copies.
Agent wiki_search_agent takes 3-step Action:
{
	name: Finish
	params: {'response': "Arthur's Magazine"}
}
Observation: Arthur's Magazine
=========wiki_search_agent finish execution. TaskPackage[ID:03244001-d72a-4b5e-b5a1-bf22bde7636d] status:
[
	completion: completed
	answer: Arthur's Magazine
]
==========
Processing:   1%|          | 1/100 [00:55<1:31:58, 55.74s/it]Agent wiki_search_agent receives the following TaskPackage:
[
	Task ID: 03244001-d72a-4b5e-b5a1-bf22bde7636d
	Instruction: The Oberoi family is part of a hotel company that has a head office in what city?
]
====wiki_search_agent starts execution on TaskPackage 03244001-d72a-4b5e-b5a1-bf22bde7636d====
Agent wiki_search_agent takes 0-step Action:
{
	name: Think
	params: {'response': 'I need to search for the Oberoi family and their associated hotel company, and then find the location of its head office.'}
}
Observation: OK
Agent wiki_search_agent takes 1-step Action:
{
	name: Wikipedia_Search
	params: {'query': 'Oberoi family hotel company'}
}
Observation: The Oberoi Group is a luxury hotel group with its head office in New Delhi, India. Founded in 1934, the company owns and operates 32 luxury hotels and two river cruise ships in 7 countries, primarily under its Oberoi Hotels & Resorts and Trident brands. The group also operates The Oberoi Centre for Learning and Development, which is regarded as one of Asia's top institutions for hospitality education.


Agent wiki_search_agent takes 2-step Action:
{
	name: Finish
	params: {'response': 'New Delhi, India'}
}
Observation: New Delhi, India
=========wiki_search_agent finish execution. TaskPackage[ID:03244001-d72a-4b5e-b5a1-bf22bde7636d] status:
[
	completion: completed
	answer: New Delhi, India
]
==========
Processing:   2%|▏         | 2/100 [01:10<51:35, 31.58s/it]  Agent wiki_search_agent receives the following TaskPackage:
[
	Task ID: 03244001-d72a-4b5e-b5a1-bf22bde7636d
	Instruction:  What nationality was James Henry Miller's wife?
]
====wiki_search_agent starts execution on TaskPackage 03244001-d72a-4b5e-b5a1-bf22bde7636d====
Agent wiki_search_agent takes 0-step Action:
{
	name: Think
	params: {'response': 'I need to search for James Henry Miller and find information about his wife to determine her nationality.'}
}
Observation: OK
Agent wiki_search_agent takes 1-step Action:
{
	name: Wikipedia_Search
	params: {'query': 'James Henry Miller'}
}
Observation: Henry Valentine Miller (December 26, 1891 – June 7, 1980) was an American novelist, short story writer and essayist. He broke with existing literary forms and developed a new type of semi-autobiographical novel that blended character study, social criticism, philosophical reflection, stream of consciousness, explicit language, sex, surrealist free association, and mysticism. His most characteristic works of this kind are Tropic of Cancer, Black Spring, Tropic of Capricorn, and the trilogy The Rosy Crucifixion, which are based on his experiences in New York City and Paris (all of which were banned in the United States until 1961). He also wrote travel memoirs and literary criticism, and painted watercolors.
Agent wiki_search_agent takes 2-step Action:
{
	name: Action:Wikipedia_Search[{"query": "James Henry Miller wife nationality"}]
	params: {}
}
Observation: "This is the wrong action to call. Please check your available action list.
Agent wiki_search_agent takes 3-step Action:
{
	name: Action:Wikipedia_Search[{"query": "Ewan MacColl"}]
	params: {}
}
Observation: "This is the wrong action to call. Please check your available action list.
Agent wiki_search_agent takes 4-step Action:
{
	name: Wikipedia_Search
	params: {'query': 'Ewan MacColl'}
}
Processing:   2%|▏         | 2/100 [01:20<1:05:29, 40.09s/it]
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.2.1\plugins\python-ce\helpers\pydev\pydevd.py", line 1500, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.2.1\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\adria\AgentLite\benchmark\hotpotqa\evaluate_hotpot_qa.py", line 170, in <module>
    f1, acc = run_hotpot_qa_agent(level=args.level, llm_name=args.llm, agent_arch=args.agent_arch, PROMPT_DEBUG_FLAG=args.debug)
  File "C:\Users\adria\AgentLite\benchmark\hotpotqa\evaluate_hotpot_qa.py", line 121, in run_hotpot_qa_agent
    response = agent(test_task_pack)
  File "C:\Users\adria\AgentLite\agentlite\agents\BaseAgent.py", line 115, in __call__
    self.execute(task)
  File "C:\Users\adria\AgentLite\agentlite\agents\BaseAgent.py", line 150, in execute
    observation = self.forward(task, action)
  File "C:\Users\adria\AgentLite\agentlite\agents\BaseAgent.py", line 238, in forward
    observation = action(**agent_act.params)
  File "C:\Users\adria\AgentLite\benchmark\hotpotqa\SearchActions.py", line 19, in __call__
    article = wikipedia.page(search_results[0])
  File "C:\Users\adria\.conda\envs\AgentLite\lib\site-packages\wikipedia\wikipedia.py", line 276, in page
    return WikipediaPage(title, redirect=redirect, preload=preload)
  File "C:\Users\adria\.conda\envs\AgentLite\lib\site-packages\wikipedia\wikipedia.py", line 299, in __init__
    self.__load(redirect=redirect, preload=preload)
  File "C:\Users\adria\.conda\envs\AgentLite\lib\site-packages\wikipedia\wikipedia.py", line 345, in __load
    raise PageError(self.title)
wikipedia.exceptions.PageError: Page id "ewan malcolm" does not match any pages. Try another id!
python-BaseException

The adapter in agents_llm.py lokks like this:

def get_llm_backend(llm_config: LLMConfig):
    llm_name = llm_config.llm_name
    llm_provider = llm_config.provider
    if llm_name in OPENAI_CHAT_MODELS:
        return LangchainChatModel(llm_config)
    if llm_name in "gpt-4o":
        llm_name = "gpt-4o"
        llm_config_dict = {
            "llm_name": llm_name,
            "temperature": 0.01,
            "context_len": 4096,
        }
        llm_config = LLMConfig(llm_config_dict)
        return AzureOpenAIChatLLM(llm_config=llm_config)

    elif llm_name in OPENAI_LLM_MODELS:
        return LangchainLLM(llm_config)
    else:
        return LangchainLLM(llm_config)
    # TODO: add more llm providers and inference APIs but for now we are using langchainLLM as the default
    # Using other LLM providers will require additional setup and configuration
    # We suggest subclass BaseLLM and implement the run method for the specific provider in your own best practices

Are we sticking to original value propostion

The initial value proposition of AgentLite was to provide a lightweight alternative agent framework with minimal code abstraction. By "minimal abstraction," I mean that users shouldn't need press F12 multiple times to view the source code and prompts to understand it. I wanted to take a moment to reflect on whether we are still adhering to the original mission of this project. Any thoughts maintainers? @JimSalesforce

Any chance you can provide a prompt template in Chinese version?

Hello, appreciate for your inspiring work. Any chance you can provide a prompt template in Chinese version?

Could you publish your eval data?

Hey Salesforce-Team,
could you please release your evaluation data for the hotpotQA dataset?

salesforceairesearch / agentlite Goto Github PK

agentlite's People

Contributors

Stargazers

Watchers

Forkers

agentlite's Issues

Recommend Projects

Recommend Topics

Recommend Org