Deployed a cluster, scaling past 2 nodes failed, now can't start any about data-center-helm-charts

[kennymacleod]

Hi @nathan-sav, thanks for filing the ticket. Could you include the logs of the 3rd pod also, please. I'm not aware of a problem scaling to 3 nodes (I just tried it myself to be sure). Did you use one of the versioned published charts, or did you package it from source?

[nathan-march]

I just blew away the cluster and recreated from scratch, ran into the pod 3 issue again. Full log is attached, pod2 booted fine again.
pod3.txt

Just using your published chart (helm install jira atlassian-data-center/jira -n jira -f values.yaml)

[bianchi2]

Can't reproduce either. @nathan-sav what K8s installation do you have? On-prem, public cloud? Also, I'd look at where the 3rd pod is scheduled, which node? I sometimes had situations on my self made K8s cluster that pods placed onto the same node can reach each other using pod IPs. Just an assumption though.

UPD. It's a different error this time. DB related.

[kennymacleod]

@nathan-sav I'm thinking you may have a DB connectivity issue here. Perhaps the database hasn't been configured with enough connections, and the 3rd Jira node is hanging trying to get some.

Please note also that we have a known issue with restarting Jira pods at the moment. This happens when the restarted pod gets a different IP address to the previous pod (which is most of the time), and Jira isn't picking up the new IP correctly. This should be fixed soon.

[nathan-march]

On prem, deploying with kubespray and using the crunchydata postgres operator with pgbouncer. Interestingly, I dropped pgbouncer here and just connected directly to postgres and now I can't reproduce this (even though I don't see any limits on pgbouncer that would cause this). I'll keep digging, but does seem to be some sort of DB issue here and you can close this :)

[bianchi2]

thanks for the update @nathan-sav, and do share your feedback :)