No disk space crashloop but pod healthy about postgres-operator

[jmckulk]

Hey @mausch, I was able to replicate the issue you are seeing with a full disk. After the disk filled up, Postgres stopped working, but the pod was still ready.

We use the Patroni GET /readiness endpoint to determine whether the pod is ready or not. If you take a look at the Patroni API docs for that endpoint, it will return 200 when "the Patroni node is running as the leader OR when PostgreSQL is up and running." In a single instance, when the database goes down, there isn't another instance to take leadership if the current leader becomes unhealthy. Like you have found, this gets into a case where the database is inaccessible, but the pods are still ready.

As it stands now, if Patroni thinks that the cluster is ready then the pods will be ready. There is likely some work we could do to augment the Patroni readiness endpoint and I will be happy to get a story in our backlog.

[mausch]

Hi, thanks for looking into this.

Shouldn't the liveness probe (rather than readiness) apply here?

The Patroni docs say:

GET /health: returns HTTP status code 200 only when PostgreSQL is up and running.

GET /liveness: returns HTTP status code 200 if Patroni heartbeat loop is properly running and 503 if the last run was more than ttl seconds ago on the primary or 2*ttl on the replica. Could be used for livenessProbe.

GET /readiness: returns HTTP status code 200 when the Patroni node is running as the leader or when PostgreSQL is up and running. The endpoint could be used for readinessProbe when it is not possible to use Kubernetes endpoints for leader elections (OpenShift).

I don't have a pgo cluster at hand to check but presumably Patroni's /liveness is already mapped to the pod liveness?