k8sgpt analyse --explains fails when using localai (401) about k8sgpt

[VaibhavMalik4187]

The --explain flag uses the AI provider configured by the user to run an analysis by communicating with the AI through the API keys. Localai doesn't use any API keys or AI and this is why you're facing this error. However, I think that the output could be improved to help the user understand the real problem here.

[sysnet4admin]

@ronaldpetty FYI
If model is proper working on

Here is the result... Even though it could not support in Korean :)

root@cp-k8s:~# k8sgpt analyze --explain -b localai -l Korean
 100% |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (2/2, 52 it/hr)           
AI Provider: localai

0 default/llama-2-7b-chat(llama-2-7b-chat)
- Error: Service has not ready endpoints, pods: [Pod/llama-2-7b-chat-55f89d58df-bwlzt], expected 1
 Error: Service has not ready endpoints, pods: [Pod/llama-2-7b-chat-55f89d58df-bwlzt], expected 1.

Solution:

1. Check if the service is running by running `kubectl get svc llama-2-7b-chat`
2. If the service is not running, run `kubectl start sfllama-2-7b-chat` to start it.
3. Wait for the service to be ready by running `kubectl wait --for=service=llama-2-7b-chat-ready`
4. Once the service is ready, check the endpoints by running `kubectl get ep llama-2-7b-chat`
1 default/llama-2-7b-chat-55f89d58df-bwlzt(Deployment/llama-2-7b-chat)
- Error: failed to generate container "288eac52810fb2072ef983c70dd57bba05cdf181524a92ff51e5ca55d4da8084" spec: failed to generate spec: failed to stat "/var/lib/kubelet/pods/60c6876c-2a05-4499-92e8-5f326a9e046e/volumes/kubernetes.io~nfs/pvc-d82ae098-9e08-4dc7-90cf-09e4b9dd79cd": stat /var/lib/kubelet/pods/60c6876c-2a05-4499-92e8-5f326a9e046e/volumes/kubernetes.io~nfs/pvc-d82ae098-9e08-4dc7-90cf-09e4b9dd79cd: stale NFS file handle
 Error: failed to stat "/var/lib/kubelet/pods/60c6876c-2a05-4499-92e8-5f326a9e046e/volumes/kubernetes.io~nfs/pvc-d82ae098-9e08-4dc7-90cf-09e4b9dd79cd": stale NFS file handle
Solution:
1. Check and update the Kubernetes version on your cluster.
2. Run `kubectl cleanup` to remove any stale state left over from previous runs.
3. Restart the kubelet to refresh the file handles.
4. Try running the `kubectl create` command again.

[ronaldpetty]

@sysnet4admin I am very new to this. Per @VaibhavMalik4187 it sounds like this would never work (unless local-ai started accepting tokens). Can you share how you set is done? I used the k8sgpt operator and localai.

[sysnet4admin]

@ronaldpetty I use just binary k8sgpt + ollama(https://github.com/ollama/ollama)
Hopefully it would understand further.

[sysnet4admin]

You could leverage this one for loading models in local env easily.
(text-generation-webui)

[JuHyung-Son]

how did you set your localai?

[liyimeng]

@ronaldpetty I use just binary k8sgpt + ollama(https://github.com/ollama/ollama) Hopefully it would understand further.

@sysnet4admin may you kindly share your backend configuration. I try to use ollama, but could not find any docs on how to set it up. Thanks a lot!

[sysnet4admin]

@JuHyung-Son @liyimeng
https://yozm.wishket.com/magazine/detail/2516/ (writer is me @sysnet4admin )

Hopefully it will help to understand for setup & moving further. (unfortunately this article wrote in Korean, but content may understand due to command is in EN).

[liyimeng]

Thanks for instantly response! @sysnet4admin I will give it a shot!

[ronaldpetty]

Thank you @sysnet4admin . I think I am starting to understand based on your tutorial. I was using this tutorial originally:

https://itnext.io/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65

I believe these issues were causing me trouble.

• localai != local-ai (the Golang project) // this really confused me until today
• the tutorial I used has an issue, it assume the service is on 8080 but it is on 80

Once I updated the cluster service port, the errors went away. However, I am still confused on basic operations of k8sgpt.

~$ k8sgpt analyze 
AI Provider: AI not used; --explain not set

0 default/broken-pod(broken-pod)
- Error: Back-off pulling image "nginx:1.a.b.c"

What does that even mean? (AI Provider not used?) Does that mean k8sgpt still does analyse even if there are no backends?

Another thing I noticed. If you have no k8sgpt resource, nothing happens (e.g. submit broken pod and never get a result). If you do have a k8sgpt, even with a broken baseURL (like I had), it still creates a result. Again, that is confusing, it implies no AI is needed. Maybe it says that somewhere but I must be missing it.

I think my issue title might need to be changed to local-ai and localai.

Final thing, I installed the operator via helm. The logs look as such:

...
Finished Reconciling k8sGPT
Creating new client for 10.43.58.77:8080
Connection established between 10.43.58.77:8080 and localhost with time out of 1 seconds.
Remote Address : 10.43.58.77:8080 
K8sGPT address: 10.43.58.77:8080
Checking if defaultbrokenpod is still relevant

It would be helpful if the operator logged who it was talking to (maybe its not talking to anyone, but then I don't understand this configuration:

~$ cat k8sgpt-localai.yaml 
apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local-ai
  namespace: default
spec:
  ai:
    enabled: true
    model: ggml-gpt4all-j
    backend: localai
    baseUrl: http://local-ai.default.svc.cluster.local:80/v1
  noCache: false
  repository: ghcr.io/k8sgpt-ai/k8sgpt
  version: v0.3.8

Here are the svc's I have.

~$ kubectl get svc
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes        ClusterIP   10.43.0.1      <none>        443/TCP    156m
local-ai          ClusterIP   10.43.90.122   <none>        80/TCP     122m
k8sgpt-local-ai   ClusterIP   10.43.58.77    <none>        8080/TCP   19m

[ronaldpetty]

I made some more progress. I got k8sgpt CLI tool working with OpenAI. I went back to LocalAI, and made a little more progress. I tried the k8sgpt CLI in a Pod, since I installed LocalAI there. It seems that I am misusing LocalAI.

root@testing:/# k8sgpt auth add --backend localai --model tinyllama --baseurl http://local-ai.default.svc.cluster.local:80/v1
localai added to the AI backend provider list

root@testing:/# k8sgpt analyze --explain -b localai --kubeconfig=.kube/config
   0% |                                                                                                                                                                                                                            | (0/1, 0 it/hr) [0s:0s]
Error: failed while calling AI provider localai: error, status code: 500, message: could not load model - all backends returned error: 23 errors occurred:
	* could not load model: rpc error: code = Canceled desc = 
	* could not load model: rpc error: code = Unknown desc = failed loading model
	* could not load model: rpc error: code = Unknown desc = failed loading model
	* could not load model: rpc error: code = Unknown desc = failed loading model
	* could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
	* could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
	* could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/tinydream. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* could not load model: rpc error: code = Unknown desc = unsupported model type /models/add (should end with .onnx)
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers-musicgen/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/mamba/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/petals/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
	* grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS

Below are the logs from LocalAI. I at least know its hitting it, but seems to be failing due to some configuration issue. I guess maybe will close this ticket and focus on LocalAI.

4:17AM INF Trying to load the model 'add' with all the available backends: llama-cpp, llama-ggml, gpt4all, bert-embeddings, rwkv, whisper, stablediffusion, tinydream, piper, /build/backend/python/transformers/run.sh, /build/backend/python/vllm/run.sh, /build/backend/python/transformers-musicgen/run.sh, /build/backend/python/sentencetransformers/run.sh, /build/backend/python/autogptq/run.sh, /build/backend/python/sentencetransformers/run.sh, /build/backend/python/coqui/run.sh, /build/backend/python/exllama2/run.sh, /build/backend/python/bark/run.sh, /build/backend/python/mamba/run.sh, /build/backend/python/diffusers/run.sh, /build/backend/python/vall-e-x/run.sh, /build/backend/python/petals/run.sh, /build/backend/python/exllama/run.sh
4:17AM INF [llama-cpp] Attempting to load
4:17AM INF Loading model 'add' with backend llama-cpp
4:17AM INF [llama-cpp] Fails: could not load model: rpc error: code = Canceled desc = 
4:17AM INF [llama-ggml] Attempting to load
4:17AM INF Loading model 'add' with backend llama-ggml
4:17AM INF [llama-ggml] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
4:17AM INF [gpt4all] Attempting to load
4:17AM INF Loading model 'add' with backend gpt4all
4:17AM INF [gpt4all] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
4:17AM INF [bert-embeddings] Attempting to load
4:17AM INF Loading model 'add' with backend bert-embeddings
4:17AM INF [bert-embeddings] Fails: could not load model: rpc error: code = Unknown desc = failed loading model
4:17AM INF [rwkv] Attempting to load
4:17AM INF Loading model 'add' with backend rwkv
4:17AM INF [rwkv] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF
4:17AM INF [whisper] Attempting to load
4:17AM INF Loading model 'add' with backend whisper
4:17AM INF [whisper] Fails: could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
4:17AM INF [stablediffusion] Attempting to load
4:17AM INF Loading model 'add' with backend stablediffusion
4:17AM INF [stablediffusion] Fails: could not load model: rpc error: code = Unknown desc = stat /models/add: no such file or directory
4:17AM INF [tinydream] Attempting to load
4:17AM INF Loading model 'add' with backend tinydream
4:17AM INF [tinydream] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/tinydream. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [piper] Attempting to load
4:17AM INF Loading model 'add' with backend piper
4:17AM INF [piper] Fails: could not load model: rpc error: code = Unknown desc = unsupported model type /models/add (should end with .onnx)
4:17AM INF [/build/backend/python/transformers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/transformers/run.sh
4:17AM INF [/build/backend/python/transformers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/vllm/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/vllm/run.sh
4:17AM INF [/build/backend/python/vllm/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vllm/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/transformers-musicgen/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/transformers-musicgen/run.sh
4:17AM INF [/build/backend/python/transformers-musicgen/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/transformers-musicgen/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/sentencetransformers/run.sh
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/autogptq/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/autogptq/run.sh
4:17AM INF [/build/backend/python/autogptq/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/autogptq/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/sentencetransformers/run.sh
4:17AM INF [/build/backend/python/sentencetransformers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/sentencetransformers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/coqui/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/coqui/run.sh
4:17AM INF [/build/backend/python/coqui/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/coqui/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/exllama2/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/exllama2/run.sh
4:17AM INF [/build/backend/python/exllama2/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama2/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/bark/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/bark/run.sh
4:17AM INF [/build/backend/python/bark/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/bark/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/mamba/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/mamba/run.sh
4:17AM INF [/build/backend/python/mamba/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/mamba/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/diffusers/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/diffusers/run.sh
4:17AM INF [/build/backend/python/diffusers/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/diffusers/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/vall-e-x/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/vall-e-x/run.sh
4:17AM INF [/build/backend/python/vall-e-x/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/vall-e-x/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/petals/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/petals/run.sh
4:17AM INF [/build/backend/python/petals/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/petals/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS
4:17AM INF [/build/backend/python/exllama/run.sh] Attempting to load
4:17AM INF Loading model 'add' with backend /build/backend/python/exllama/run.sh
4:17AM INF [/build/backend/python/exllama/run.sh] Fails: grpc process not found: /tmp/localai/backend_data/backend-assets/grpc/build/backend/python/exllama/run.sh. some backends(stablediffusion, tts) require LocalAI compiled with GO_TAGS