Using davinci-003 makes results worse? about dspy

[okhat]

Thanks for testing this!I haven’t worked with 003 in any serious capacity before, but there are academic results that show it is in fact worse for academic tasks than 002. It could still be better for casual use, open-ended generation, or chatbots, etc. Happy to find references if needed.

That said, I absolutely wouldn’t read much at all into the score on the 13 examples. A difference of 4 matches is too small to be representative in a statistical sense. The examples are only given for illustration, as the notebook says, rather than a benchmark.

But it might still be interesting to study the ways in which the new model makes mistakes not made by the older model.