504 Gateway Time-out about kubeadm-bootstrap

[zonca]

please give us more info.

sudo kubectl get pods --all-namespaces
sudo helm ls

if any pod is failing, run describe and logs on that.

[julienchastang]

$ sudo kubectl get pods --all-namespaces

NAMESPACE     NAME                                                    READY     STATUS    RESTARTS   AGE
default       rook-agent-2prs8                                        1/1       Running   0          3h
default       rook-agent-fctrt                                        1/1       Running   0          3h
default       rook-operator-6886996bcb-bq9nn                          1/1       Running   0          3h
kube-system   etcd-tg-atm160027-master-node                           1/1       Running   0          4h
kube-system   kube-apiserver-tg-atm160027-master-node                 1/1       Running   0          4h
kube-system   kube-controller-manager-tg-atm160027-master-node        1/1       Running   0          4h
kube-system   kube-dns-6f4fd4bdf-gbjn7                                3/3       Running   0          4h
kube-system   kube-flannel-ds-6kclb                                   1/1       Running   1          4h
kube-system   kube-flannel-ds-kgm2f                                   1/1       Running   0          4h
kube-system   kube-proxy-mjn2v                                        1/1       Running   0          4h
kube-system   kube-proxy-mt549                                        1/1       Running   0          4h
kube-system   kube-scheduler-tg-atm160027-master-node                 1/1       Running   0          4h
kube-system   tiller-deploy-69cb6984f-m6jw7                           1/1       Running   0          4h
support       support-nginx-ingress-controller-pc5qw                  1/1       Running   0          4h
support       support-nginx-ingress-controller-v6bg4                  1/1       Running   1          4h
support       support-nginx-ingress-default-backend-cb84895fb-s6bhs   1/1       Running   0          4h

$ sudo helm ls
sudo: unable to resolve host tg-atm160027-master-node
NAME            REVISION        UPDATED                         STATUS          CHART           NAMESPACE
insipid-sponge  1               Fri Jun 22 18:48:55 2018        DEPLOYED        rook-v0.7.1     default
support         1               Fri Jun 22 17:56:54 2018        DEPLOYED        support-0.1.0   support

[zonca]

check into the logs of the nginx controllers and backend, is this Kubernetes 1.9?

[julienchastang]

I'm going down this road again. I cannot recall why I did not respond to this last comment on June 23. I am seeing the same problem. This is v1.9.2. Where are the logs in question? master or worker node?

ping @craig-willis

[craig-willis]

This is likely a problem with the security group configuration in OpenStack. You need to enable traffic between nodes in your Kubernetes cluster. The https://github.com/nds-org/kubeadm-terraform/ process creates a security group (which is conveniently already in place on your TACC cluster). Applying that security group to the nodes, you should now see for both hosts:

$ curl localhost
default backend - 404

[julienchastang]

Thanks for your help. I can confirm the security group defined here resolves the issue. Though I cannot figure out exactly what it is about that security group that makes it work. (Not everyone is going to want to use terraform.) It could be the opening up of tcp and udp ports 1 through 65535 to "self", but I don't know what "self" is since it is undocumented in openstack and also here .

[craig-willis]

self is an argument in the Terraform OpenStack provider. According to my colleague @bodom0015, this equates to creating a rule with "Remote" set to "Security Group" and the "Security Group" set to the current group name (test in the attached screenshot):

From my understanding, this says that any node in the security group can communicate with any other node in that group. This is used instead of an explicit CIDR range.

[julienchastang]

That makes sense. I do something similar with my openstack security groups with --remote-ip 10.0.0.0/24 to limit traffic internally, but this may be a better, more "openstack" way.

[julienchastang]

In summary open ports 1 through 65535, but limit traffic internal to the security group itself. I.e. from openstack,

openstack security group rule create kube --protocol udp --dst-port 1:65535 --remote-group kube
openstack security group rule create kube --protocol tcp --dst-port 1:65535 --remote-group kube

in addition to whatever outward ports you need open i.e., 22, 80, 443, 6443.

I think we can close this issue.