Buggy behavior after failed health check recover about kong

[StarlightIbuki]

The behavior sounds expected to me. The health check status does not update immediately, and the passive health checker cannot predict if the next request will succeed. Could you elaborate?

[mhkarimi1383]

@StarlightIbuki
Hi
But after the interval passes it should recover, but it doesn't.
Also clearing Kong cache via Admin API fixes the issue

[mhkarimi1383]

It happens when we have a rolling update on our K8s Deployment

[StarlightIbuki]

@mhkarimi1383 Is the upstream failing in a predictable or manipulatable manner? So that you are sure that the status is not reflecting the fact?

[mhkarimi1383]

@StarlightIbuki
Yes
I have sent a request to that pod and monitor that health check endpoint using a blackbox exporter pointing to it's k8s service

[StarlightIbuki]

@mhkarimi1383 Could you share the config that you are using?

[mhkarimi1383]

@StarlightIbuki

        upstream:
          healthchecks:
            active:
              healthy:
                interval: 5
                successes: 3
              type: tcp
              unhealthy:
                tcp_failures: 1
                interval: 5

Here is my KongIngress spec

[StarlightIbuki]

5s seems a short interval. How long do you wait before inspecting the status?

[mhkarimi1383]

@StarlightIbuki
About 5 minutes

[StarlightIbuki]

I still do not really understand the reproduction steps. When the health checker reports green and you get 503, what real status are you expecting?

[mhkarimi1383]

I still do not really understand the reproduction steps. When the health checker reports green and you get 503, what real status are you expecting?

Yes,
Kong said the project is unhealthy but it is healthy, clear king cache fixes the problem

[StarlightIbuki]

I still do not really understand the reproduction steps. When the health checker reports green and you get 503, what real status are you expecting?

Yes, Kong said the project is unhealthy but it is healthy, clear king cache fixes the problem

Sorry, but let's confirm if my understanding is correct: for step 4, we configure the upstream to back to work again, and we will observe the health checker reporting unhealthy condition?

[mhkarimi1383]

@StarlightIbuki Yes

[github-actions]

This issue is marked as stale because it has been open for 14 days with no activity.