Message prompt for image should support OpenAI syntax or warn user about ai

[lgrammel]

Are you using typescript or javascript? Can you provide an example of the input that resulted in this error?

[castortech]

I am using TS.

Here is a message causing the issue:

[
  {
    role: "system",
    content: "Today is 8/16/2024.\n\nUser Instructions:\nYou are a friendly, helpful AI assistant.",
  },
  {
    role: "user",
    content: [
      {
        type: "text",
        text: "explain image",
      },
      {
        type: "image_url",
        image_url: {
          url: "data:image/jpeg;base64,/9j/4aJORXhpZgAASUkqAAgAAAANAAABAwABAAAAAAwAAAEBAwABAAAA8A8AAA8BAgAHAAAAqgAAABABAgAMAAAAsQAAABIBAwABAAAAAQAAABoBBQABAAAAvQAAABsBBQABAAAAxQAAACgBAwABAAAAAgAAADEBAgAVAAAAzQAAADIBAgAUAAAA4gAA...=",
        },
      },
    ],
  },
]

and the resulting error:

[TypeError: Cannot read properties of undefined (reading 'type')]
Error in chat/openai: TypeError: Cannot read properties of undefined (reading 'type')
    at eval (webpack-internal:///(rsc)/./node_modules/@ai-sdk/openai/dist/index.mjs:41:26)
    at Array.map (<anonymous>)
    at convertToOpenAIChatMessages (webpack-internal:///(rsc)/./node_modules/@ai-sdk/openai/dist/index.mjs:39:28)
    at OpenAIChatLanguageModel.getArgs (webpack-internal:///(rsc)/./node_modules/@ai-sdk/openai/dist/index.mjs:254:17)
    at OpenAIChatLanguageModel.doStream (webpack-internal:///(rsc)/./node_modules/@ai-sdk/openai/dist/index.mjs:380:37)
    at fn (webpack-internal:///(rsc)/./node_modules/ai/dist/index.mjs:2942:35)
    at eval (webpack-internal:///(rsc)/./node_modules/ai/dist/index.mjs:328:28)
    at Object.startActiveSpan (webpack-internal:///(rsc)/./node_modules/ai/dist/index.mjs:257:14)
    at recordSpan (webpack-internal:///(rsc)/./node_modules/ai/dist/index.mjs:326:17)
    at eval (webpack-internal:///(rsc)/./node_modules/ai/dist/index.mjs:2914:15)

[lgrammel]

@castortech I'm getting a TS error when I try to do this:

Did this error show up for you? Or was it a situation where any was used?

(we might need to improve our internal checks regardless of typing bc of any)

[castortech]

I am not forming the message manually, and a bit closer to any

Here is the type info:

let finalMessages: {
    role: string;
    content: string | ({
        type: string;
        image_url: {
            url: string;
        };
        image?: undefined;
    } | {
        type: string;
        image: string;
        image_url?: undefined;
    } | {
        type: string;
        text: string;
    })[];
}[]

that was made to be multi-provider but mostly OpenAI and that we are working to convert fully to Vercel AI (prior was using legacy provider only to stream response).