About generate_program regular expression about cumulative-reasoning

[yjqqqaq]

Thank you for your question. The main reason we added special judge here is to prevent unknown format errors when calling the API. In fact, you'll find in the prompts of 'generate_program' that we expect 'Next Step' to be an equation composed solely of numbers and equals signs, which makes it more convenient for us to handle. When we use GPT-4 as the model, it is able to correctly understand our needs and almost never produces additional descriptive text that would cause this line of judgment to fail. So we added this judgment simply to avoid unexpected errors during the API execution of GPT-4.

We guess the problem you encountered may be due to GPT-3.5-Turbo, which interprets the prompt in a slightly different way than GPT-4, resulting in unnecessary extra description words in the output. We need to point out that if the output format of the "Next Step" is uncertain and contains additional descriptive text, extracting the "Next Step" equation we need is a non-trivial task. Therefore, in such situations, we choose to ignore this API call and regenerate it.

We will take your question seriously, and in future updates, we will try to solve Game 24 using GPT-3.5-Turbo, and attempt to provide new prompts that are compatible with both GPT-3.5-Turbo and GPT-4.