Tool function leak on AI answer about ai

[lgrammel]

@agusmoles it's a wild guess, but one thing that could throw off the LLM is the mix of english and french. i often got better results by going all in for a given language. This unfortunately means that you property names, e.g.tnaInDecimals might need to be french, or you could at least add a French description to the variable in the schema. Maybe that helps.

[jeremyphilemon]

@agusmoles Thanks for reporting!

This seems like it's an issue with the generation where the model's output is wrong. Were you expecting it to call a tool but instead it printed out the python code to perform the calculation?

[agusmoles]

@jeremyphilemon yes I know there is too little to do since its pretty difficult to replicate and/or report since the whole context is given from an AI model, so feel free to close if you think there is nothing vercel could do to avoid this kind of "leaks".

Regarding the prompts, the questions you can see in the chat are pretty straightforward, the first asks for an economic TNA (APR in US) which returns a tool call, the second one is a question about the capitalization of the interests earned from the investment (which is given in the system attribute of the model for context) and the third one was intended to also execute another tool function which aims to convert the TNA to a TEA (Effective Annual Rate), but it threw that python code.

It didnt happen before with the same tool or others, it was just that unique time, thats why also the probability of replicating is pretty low but I wanted to report just in case. 😄

Thanks!

[agusmoles]

@jeremyphilemon exactly that, yes

[jeremyphilemon]

@agusmoles Thanks for clarifying!

It seems like it's an issue with the system prompt, schema annotation, or the language model itself, so there's very little we can do as an sdk to prevent this since it's application specific. Any chance you can share more details on the prompt you used that could've led the model to generate a python code instead of calling a tool?