llm.api_key is not found in auto eval about taskweaver

[zhangxu0307]

Hi, @saakethch ! Could you please provide more detailed information about the error? We are uncertain whether the issue is caused by the auto_eval configuration or the taskweaver configuration. If possible, kindly share the call stack and error logs with us. Thank you.

[saakethch]

Hi, this the error i get :

Traceback (most recent call last):
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver_eval.py", line 126, in <module>
    score_list = auto_evaluate_for_taskweaver(eval_case_file_path, interrupt_threshold=None)
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver_eval.py", line 32, in auto_evaluate_for_taskweaver
    app = TaskWeaverApp(app_dir=app_dir, config=config_var)        
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\app\app.py", line 43, in __init__
    self._init_app_modules()
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\app\app.py", line 91, in _init_app_modules    
    self.app_injector.get(LLMApi)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 975, in get
    result = provider_instance.get(self)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 264, in get
    return injector.create_object(self._cls)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 998, in create_object    
    self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 1031, in call_with_injection
    dependencies = self.args_to_inject(
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 1079, in args_to_inject  
    instance: Any = self.get(interface)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 91, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 975, in get
    result = provider_instance.get(self)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 264, in get
    return injector.create_object(self._cls)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 998, in create_object    
    self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
  File "C:\Users\15512\AppData\Local\Programs\Python\Python310\lib\site-packages\injector\__init__.py", line 1040, in call_with_injection
    return callable(*full_args, **dependencies)
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\config\module_config.py", line 13, in __init__
    self._configure()
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\llm\__init__.py", line 27, in _configure      
    self.api_key = self._get_str(
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\config\module_config.py", line 25, in _get_str    return self.src.get_str(self._config_key(key), default)        
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\config\config_mgt.py", line 135, in get_str   
    val = self._get_config_value(var_name, "str", default_value)   
  File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\config\config_mgt.py", line 96, in _get_config_value
    raise ValueError(f"Config value {var_name} is not found")      
ValueError: Config value llm.api_key is not found

main method:

    # run single case
    eval_case_file_path = "auto_eval\cases\complicated_task_stock_forecasting.yaml"
    score_list = auto_evaluate_for_taskweaver(eval_case_file_path, interrupt_threshold=None)
    for idx, score, normalized_score in score_list:
        print(f"Round-{idx} score: {score}, normalized score: {normalized_score}")

    # run batch cases
    result_file_path = "ttt.csv"
    case_file_dir = "auto_eval\cases"
    batch_auto_evaluate_for_taskweaver(
        result_file_path,
        case_file_dir,
        flush_result_file=False,
        interrupt_threshold=None,
    )```
    
    I've mentioned the llm key at both places

[zhangxu0307]

Have you moved taskweaver folder to auto_eval?
d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\auto_eval\taskweaver\config\module_config.py.
In this case, Taskweaver would not find its config file correctly. You only need to complete the evalutaion_config.json file and taskweaver_config.json file, then run the evaluation script.

[saakethch]

I still get the same error:

taskweaver_eval.py placed in root where the taskweaver source folder exists.

if __name__ == "__main__":
    # run single case
    eval_case_file_path = "auto_eval\cases\init_say_hello.yaml"

outputs error: 
    
File "d:\HolonicSpace\space\mmmu\TaskWeaver-main\TaskWeaver-main\taskweaver\config\config_mgt.py", line 96, in _get_config_value      
    raise ValueError(f"Config value {var_name} is not found")
ValueError: Config value llm.api_key is not found

I have my api key in auto_eval\evaluator_config.json

[zhangxu0307]

Plz refer to readme file to run the auto evalutaion script. Thanks.

[saakethch]

Thanks, That helped.

Now the auto_eval runs but at the very end I get this :
AttributeError: module 'openai' has no attribute 'error'

I have installed taskweaver in a virtualenv

[zhangxu0307]

Thanks, That helped.

Now the auto_eval runs but at the very end I get this : AttributeError: module 'openai' has no attribute 'error'

I have installed taskweaver in a virtualenv

Could you provide the detailed error logs? Thanks.

[zhangxu0307]

We will close this issue because of no more updating.