AI-Proxy plugin: "An unexpected error occurred" when upstream URL is missing port/path about kong

[Water-Melon]

It is tracked by KAG-4312

[rohitrsh]

Hey @Water-Melon, I don't have access to Kong JIRA. How can I get updates on KAG-4312? I am also facing same issue with self hosted llama2 model

[chronolaw]

#12903 may fix this issue, could you check it? thanks.

[dascole]

@chronolaw I have tested against this branch but it throws another 500

{
	"message": "An unexpected error occurred",
	"request_id": "b1dc385ad051903a0a30441d9b7dc97d"
}

2024/05/06 10:24:52 [error] 2660914#0: *12887 [kong] init.lua:405 [ai-proxy] ./kong/llm/drivers/llama2.lua:267: bad argument #1 to 'string_gsub' (string expected, got nil), client: 127.0.0.1, server: kong, request: "POST /echo HTTP/1.1", host: "localhost:8000", request_id: "b1dc385ad051903a0a30441d9b7dc97d"

When no path is specified parsed_url.path is nil

2024/05/06 10:24:52 [notice] 2660914#0: *12887 [lua] llama2.lua:265: configure_request(): parsed_url = {
  authority = "myserver:11434",
  host = "myserver",
  port = "11434",
  scheme = "http"
}

$ kong version
3.7.0

$ git branch
* feat/streamline_ai_proxy

[chronolaw]

Thank you @dascole , we have recorded it in the ticket KAG-4312, we are still trying to fix it.

[tysoekong]

Hi everyone; many apologies for this bug - I had originally fixed it and I knew it was coming in 3.7.0, so I wasn't watching this issue, and now during refactoring the URL parser for fixing OLLAMA token streaming I had broken it again. This is my fault here.

I have opened the suggested fix and tested it again: #12998 but we may be frozen for 3.7.0 features now, so it might roll into fixes for 3.7.1 instead.

See the linked pull request for updates on this one.