Problem with PairwiseAutoAnnotator about alpaca_farm

[YannDubs]

Hi @langhaobeijing, sorry for the delay in answer!

I just tested and it works for me. Cany you try to update alpaca_eval pip install -U alpaca_eval

[langhaobeijing]

Hi @YannDubs,

Thanks for your time! The code can run well in interactive mode of python, but cannot run  successfully in script mode.

[YannDubs]

thanks @langhaobeijing , I was able to reproduce it. This is very strange. I'll let you know when I solve that problem

[YannDubs]

Hi @langhaobeijing the issue is likely that you are running a script without a if __name__ == '__main__': block which doesn't work if you use multiprocessing (which is the default for openai completion function) because every process will try to import the file that called it, which will cause the script to rerun.

You have two options, to avoid this issue:

1. You can set OPENAI_MAX_CONCURRENCY=1 to remove multi processing
2. you can run your script with a if __name__ == '__main__': block like in the following

from alpaca_farm.auto_annotations import PairwiseAutoAnnotator
import json


def run():
    with open("examples/data/outputs_pairs.json") as f:
        outputs_pairs = json.load(f)
        annotator = PairwiseAutoAnnotator(annotators_config="test", caching_path=None)
        annotated = annotator.annotate_pairs(outputs_pairs)

if __name__ == '__main__':
    run()