Graceful shutdown for talaria about talaria

[atlas-comstock]

The restart issue is fixed, but we can still discuss this.

[tardunge]

When we do a rollout restart or k8s decided to restart/move the workload to a different node(typical k8s operations), it is terminating the pod with a grace period of 30s(default and can be customised in the pod spec). From what i understand, k8s changes the state of pod from RUNNING to TERMINATING, this event

1. detaches the pod from its load balancers and there won't be any incoming traffic to the containers from this point.
2. signals a SIGTERM to the containers.
   We are handling this event here(https://github.com/talariadb/talaria/blob/master/main.go#L75)
   If we want to preserve the data in memory and perform the compaction job, we need to do it here. If these jobs take more time than the one defined in terminationGracePeriodSeconds, k8s will send a SIGKILL signal to all the containers.

This simplifies our graceful-shutdown steps.

1. close gossip(Done)
2. close the gRPC server gracefully.(Done).
   https://github.com/talariadb/talaria/blob/master/internal/server/server.go#L152
   These two tasks need to be done here before we close all tables.
   a. Maybe waiting for a period of 5-10 sec for the in-memory data to be flushed to disk, or listen to Append() event.
   b. Once this is done, we can force a single compaction task, stopping the compaction scheduler first.

[atlas-comstock]

@tardunge Hi, I just found that in fact, we have full support on graceful shutdown already: https://github.com/talariadb/talaria/pull/109/files#diff-2873f79a86c0d8b3335cd7731b0ecf7dd4301eb19a82ef7a1cba7589b5252261R76

Tbh it does not work in our envs; then I investigate why it won't work.
In short, the graceful shutdown has never worked before, whether grpc close or sqs close or compact close, because the program will exit when ctx is closed/canceled.

[tardunge]

fixed at #110 (comment)