hkust-nlp / agentboard Goto Github PK

如题，reactagent版本好像不太对. 除了tools以外，其他的react版本会上传吗？（而且貌似reactagent要改一下，改成用正则表达式抽取的形式）

Amazing work here! Having tests for agents is super valuable.

Just a note to encourage contributions from other developers -- it would help to have more typing, as well as base classes for things.

Trying to wrap llama-index in this framework has been slightly confusing, since there are no types and I have to read a ton of code and debug line-by-line to see what the inputs to things are :)

python agentboard/eval_main.py --cfg-path eval_configs/main_results_all_tasks.yaml --tasks scienceworld --model gpt-4 --log_path ./results/gpt-4 --project_name evaluate-gpt-4 \

I try to test the tasks with GPT-4, but get the following errors:

File "/home/ycchen/miniconda3/envs/Agentboard/lib/python3.8/os.py", line 675, in getitem
raise KeyError(key) from None
KeyError: 'OPENAI_API_TYPE'

jericho
"Interaction Turn 17": {
"Action": "Push other to sword",
"Observation": "(the gleaming silver sword to the gleaming silver sword) Dropped. You pass out due to lack of oxygen and never wake. *** You have died *** In that game you scored 0 out of a possible 90, in 10 turns, earning you the rank of Primordial Ooze Candidate. Would you like to RESTART, RESTORE a saved game, give the FULL score for that game or QUIT",
"Progress Rate": 0
}
SR会给1

Have you or can you evaluate Google Gemini Pro to put as comparison?

{
"id": 3,
"task_name": "webshop",
"goal": "i need gluten free vegetarian smoked peppered bacon - 4 ounce (pack of 2), and price lower than 60.00 dollars",
"difficulty": "easy",
"is_done": false,
"progress_rate": 0.9166666666666666,
"grounding_acc": 1.0,
"score_change_record": [
[
1,
0.3333333333333333
],
[
2,
0.6666666666666666
],
[
4,
0.9166666666666666
]
],
"trajectory": {
"Interaction Turn 0": {
"Goal": "i need gluten free vegetarian smoked peppered bacon - 4 ounce (pack of 2), and price lower than 60.00 dollars",
"Observation": "WEB PAGE: {Let us begin shopping \n[Search]}",
"Action": "reset[]",
"Progress Rate": 0.0
},
"Interaction Turn 1": {
"Action": "search[gluten free vegetarian smoked peppered bacon 4 ounce pack of 2 price < 60.00]",
"Observation": "WEB PAGE: {\n[Back to Search] \nPage 1 (Total results: 15) \n[Next >] \n[B06Y96MXJV] \nSmoked Bacon Sea Salt 3-Pack: Smoked Bacon Chipotle, Smoked Bacon and Onion, Smoked Peppered Bacon - All-Natural Sea Salts Slowly Smoked Over Alderwood - No Gluten, No MSG, Non-GMO (12 total oz.) \n$29.99 \n[B06Y96N1KG] \nSpicy Hot Pepper Sea Salt 3-Pack: Ghost Pepper, Jalapeno, Habanero - All Natural, Delicious Blends of Peppers & Sea Salt - Gluten-Free, Kosher, No MSG, Non-GMO (12 total oz.) \n$29.99 \n[B07GJTKYJQ] \nLouisville Vegan Jerky - 5 Flavor Variety Pack, Non-GMO Soy Protein, Gluten-Free (Black Pepper, Buffalo Dill, Pepperoni, Maple Bacon, and Carolina BBQ, 3 Ounces) \n$42.99 }",
"Progress Rate": 0.3333333333333333
},
"Interaction Turn 2": {
"Action": "click[B06Y96MXJV]",
"Observation": "WEB PAGE: {\n[Back to Search] \n[< Prev] \nflavor [cocktail salts][dessert toppers][exotic][finishing salts][foodie gift][french blends][french classics][fruits & flowers][garlic lover's][gourmet grillmaster][hawaiian][herb garden][i love you & food][italian classics][italian flavors][keto smoked][mediterranean diet][natural salts][olive oil dippers][pacific cooking][paleo][pegan smoked][pizza salts][smoked #1][smoked bacon][smoked classics][smoked grillmaster][spicy chili peppers][steak rubs][summer fruits][taco seasonings][the pepper][truffle][tuxedo][vegan smoked][vegetarian smoked]\nsize [4 ounce (pack of 3)][3 piece assortment][4 ounce (pack of 2)][4 ounce (pack of 6)][4 ounce (3 count)][4 ounce (6 count)]\nSmoked Bacon Sea Salt 3-Pack: Smoked Bacon Chipotle, Smoked Bacon and Onion, Smoked Peppered Bacon - All-Natural Sea Salts Slowly Smoked Over Alderwood - No Gluten, No MSG, Non-GMO (12 total oz.) \nPrice: $29.99 \nRating: N.A. \n[Description] \n[Features] \n[Reviews] \n[Attributes] \n[Buy Now] }",
"Progress Rate": 0.6666666666666666
},
"Interaction Turn 3": {
"Action": "click[4 ounce (pack of 2)]",
"Observation": "WEB PAGE: {You have clicked 4 ounce (pack of 2).}",
"Progress Rate": 0.6666666666666666
},
"Interaction Turn 4": {
"Action": "click[Buy Now]",
"Observation": "WEB PAGE: {Result: [False]}",
"Progress Rate": 0.9166666666666666
}
}
}

{
"id": 3,
"task_name": "webshop",
"goal": "i need gluten free vegetarian smoked peppered bacon - 4 ounce (pack of 2), and price lower than 60.00 dollars",
"difficulty": "easy",
"is_done": false,
"progress_rate": 0.75,
"grounding_acc": 1.0,
"score_change_record": [
[
1,
0.3333333333333333
],
[
2,
0.6666666666666666
],
[
4,
0.75
]
],
"trajectory": {
"Interaction Turn 0": {
"Goal": "i need gluten free vegetarian smoked peppered bacon - 4 ounce (pack of 2), and price lower than 60.00 dollars",
"Observation": "WEB PAGE: {Let us begin shopping \n[Search]}",
"Action": "reset[]",
"Progress Rate": 0.0
},
"Interaction Turn 1": {
"Action": "search[gluten free vegetarian smoked peppered bacon 4 ounce pack of 2 price < 60.00]",
"Observation": "WEB PAGE: {\n[Back to Search] \nPage 1 (Total results: 15) \n[Next >] \n[B06Y96MXJV] \nSmoked Bacon Sea Salt 3-Pack: Smoked Bacon Chipotle, Smoked Bacon and Onion, Smoked Peppered Bacon - All-Natural Sea Salts Slowly Smoked Over Alderwood - No Gluten, No MSG, Non-GMO (12 total oz.) \n$29.99 \n[B06Y96N1KG] \nSpicy Hot Pepper Sea Salt 3-Pack: Ghost Pepper, Jalapeno, Habanero - All Natural, Delicious Blends of Peppers & Sea Salt - Gluten-Free, Kosher, No MSG, Non-GMO (12 total oz.) \n$29.99 \n[B07GJTKYJQ] \nLouisville Vegan Jerky - 5 Flavor Variety Pack, Non-GMO Soy Protein, Gluten-Free (Black Pepper, Buffalo Dill, Pepperoni, Maple Bacon, and Carolina BBQ, 3 Ounces) \n$42.99 }",
"Progress Rate": 0.3333333333333333
},
"Interaction Turn 2": {
"Action": "click[B06Y96MXJV]",
"Observation": "WEB PAGE: {\n[Back to Search] \n[< Prev] \nflavor [cocktail salts][dessert toppers][exotic][finishing salts][foodie gift][french blends][french classics][fruits & flowers][garlic lover's][gourmet grillmaster][hawaiian][herb garden][i love you & food][italian classics][italian flavors][keto smoked][mediterranean diet][natural salts][olive oil dippers][pacific cooking][paleo][pegan smoked][pizza salts][smoked #1][smoked bacon][smoked classics][smoked grillmaster][spicy chili peppers][steak rubs][summer fruits][taco seasonings][the pepper][truffle][tuxedo][vegan smoked][vegetarian smoked]\nsize [4 ounce (pack of 3)][3 piece assortment][4 ounce (pack of 2)][4 ounce (pack of 6)][4 ounce (3 count)][4 ounce (6 count)]\nSmoked Bacon Sea Salt 3-Pack: Smoked Bacon Chipotle, Smoked Bacon and Onion, Smoked Peppered Bacon - All-Natural Sea Salts Slowly Smoked Over Alderwood - No Gluten, No MSG, Non-GMO (12 total oz.) \nPrice: $29.99 \nRating: N.A. \n[Description] \n[Features] \n[Reviews] \n[Attributes] \n[Buy Now] }",
"Progress Rate": 0.6666666666666666
},
"Interaction Turn 3": {
"Action": "click[4 ounce (pack of 2)]",
"Observation": "WEB PAGE: {You have clicked 4 ounce (pack of 2).}",
"Progress Rate": 0.6666666666666666
},
"Interaction Turn 4": {
"Action": "click[Buy Now]",
"Observation": "WEB PAGE: {Result: [False]}",
"Progress Rate": 0.75
}
}
}

同时sample两次，过程一样，结果却不一样？