ERROR The OpenSearch REST endpoint has NOT become accessible in the expected time; exiting about viya4-monitoring-kubernetes

[dougmanoel]

@gsmith-sas has helped me out to solve this by removing old logging again, removing the newly created PVCs and then rerunning the deploy script.

@gsmith-sas could you just help me to clarify why one of the fb pods won't be Running? It complaning about insufficient CPU but node has plenty of resources to offer.

kubectl -n logging get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
eventrouter-6956d75996-hwh8z 1/1 Running 0 16h 10.244.2.39 aks-system-XX-vmss000001
v4m-es-exporter-5d795dfcb4-jmsg5 1/1 Running 0 16h 10.244.2.42 aks-system-XX-vmss000001
v4m-fb-55dsp 0/1 Pending 0 6h7m
v4m-fb-8t4tq 1/1 Running 0 16h 10.244.1.33 aks-system-XX-vmss000000
v4m-fb-nz2wd 1/1 Running 0 16h 10.244.5.9 aks-connect-XX-vmss000004
v4m-fb-p4rkg 1/1 Running 0 16h 10.244.6.14 aks-cas-XX-vmss000004
v4m-fb-pn9gn 1/1 Running 0 16h 10.244.3.19 aks-stateless-XX-vmss000004
v4m-fb-shlss 1/1 Running 0 16h 10.244.7.55 aks-compute-XX-vmss000004
v4m-fb-vrddh 1/1 Running 0 16h 10.244.2.43 aks-system-XX-vmss000001
v4m-osd-d569b45fb-njzkk 1/1 Running 0 16h 10.244.2.44 aks-system-XX-vmss000001
v4m-search-0 1/1 Running 0 16h 10.244.2.40 aks-system-XX-vmss000001
v4m-search-1 1/1 Running 0 16h 10.244.1.32 aks-system-XX-vmss000000
v4m-search-2 1/1 Running 0 16h 10.244.2.41 aks-system-XX-vmss000001

kubectl -n logging describe pod v4m-fb-55dsp
.
.
.
Events:
Type Reason Age From Message

---

Warning FailedScheduling 162m default-scheduler 0/7 nodes are available: 1 Insufficient cpu, 6 node(s) didn't match Pod's node affinity/selector.

kubectl -n logging top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
aks-cas-XX-vmss000004 244m 1% 2954Mi 1%
aks-compute-XX-vmss000004 159m 2% 2808Mi 4%
aks-connect-XX-vmss000004 122m 6% 2481Mi 19%
aks-stateful-XX-vmss000004 1789m 22% 33540Mi 57%
aks-stateless-XX-vmss000004 1382m 17% 18710Mi 32%
aks-system-XX-vmss000000 1019m 13% 12989Mi 46%
aks-system-XX-vmss000001 391m 5% 14804Mi 53%

• Also, can the remaining PVCs from old logging safely be deleted?
  kubectl -n logging get pvc
  NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
  data-v4m-es-data-0 Bound pvc-ad124dc4-2d45-439f-8379-7e7a9cd6ff09 30Gi RWO default 342d
  data-v4m-es-data-1 Bound pvc-5c4340e1-d900-4691-8279-8baf6b970770 30Gi RWO default 342d
  data-v4m-es-data-2 Bound pvc-1a55ceed-b0a6-4c55-8ede-3fb9852c0d62 30Gi RWO default 342d
  data-v4m-es-master-0 Bound pvc-c513cea0-bd82-42f3-b884-d3061820d7ee 8Gi RWO default 342d
  data-v4m-es-master-1 Bound pvc-51081153-ec45-46f1-a435-f0dd79df93d4 8Gi RWO default 342d
  data-v4m-es-master-2 Bound pvc-8d3f5d46-ec64-4095-a815-a6cdfd1869ee 8Gi RWO default 342d
  v4m-search-v4m-search-0 Bound pvc-c9eda1ca-23de-4611-9482-b436e3628eda 30Gi RWO default 16h
  v4m-search-v4m-search-1 Bound pvc-f3da059c-9ef5-449d-82a1-7d4b8c08fd8c 30Gi RWO default 16h
  v4m-search-v4m-search-2 Bound pvc-9cd2c52d-4401-46eb-a572-6ed41e5d6995 30Gi RWO default 16h

[gsmith-sas]

As @dougmanoel indicated, we were able to resolve the original problem. That problem was caused by an incorrect host name in the $USER_DIR/logging/user-values-opensearch.yaml file.

I'm following up via e-mail with @dougmanoel on the new unscheduled Fluent Bit pod issue he mentioned...which is a separate issue.

I am going to close this issue (since the original problem is resolved).