Avoid "Service unavailable" about dandi-hub

[asmacdo]

meanwhile -- is there an easy way to provide some default page/something which would instead say "Service unavailable due to X. Please try again in Y minutes. If still unavailable, check if a known issue on https://github.com/con/upptime/issues?q=is%3Aissue+hub+is%3Aopen . If not - file a new one" or alike?

When this flake is happening (IIRC) the hub pod is getting restarted. Typically for a highly available application, we'd be running 2 hub pods, as soon as hub 1 goes down, hub 2 picks up the traffic. However, since the hub pod is controlled by the jupyterhub helm chart .(and it would cost extra). I don't think thats how we want to handle it.

There might be something we can do with the Amazon side, maybe we could setup a health check and a default page during an outage? From a super quick search and gpt, I think it would work like this: health check fails, R53 changes the routing to a default page, then DNS would have to propagate, then the user gets the helpful message. But then, when the hub comes back up, I guess it would do the same thing in reverse, also with a propagation delay, so it could possibly extend the downtime?

[yarikoptic]

But can someone remind me why this 503 is happening at all (i.e. what stops that pod) or is that by design (non persistent, then why so)?
If we do restart upon hitting 503, could we partially mitigate (i.e. shorten duration and thus potentially make less likely to be hit) but hitting the service with "hearbeat" e.g. every second?

[asmacdo]

But can someone remind me why this 503 is happening at all (i.e. what stops that pod) or is that by design (non persistent, then why so)? If we do restart upon hitting 503,

I haven't looked into it since the problem will "just go away" soon, but I think satra is aware of the reason?

could we partially mitigate (i.e. shorten duration and thus potentially make less likely to be hit) but hitting the service with "hearbeat" e.g. every second?

I don't understand what you mean. As soon as the pod fails, the k8s deployment restarts the pod. (I'm assuming the 503 is the result of that pod being deleted and not whatever the error is). So I dont see how hitting the service every second could help?

[yarikoptic]

I probably have just misread your

When this flake is happening (IIRC) the hub pod is getting restarted.

as sensing the service and causing 503 is the event which triggers a new pod to start, not that it was already "getting started" before that event.