K3S with AWS VPC CNI breaks Pod communication #9716 about amazon-vpc-cni-k8s

[orsenthil]

Does VPC CNI (aws-node pod) ever get into a ready state? Are you using k3s on a EC2 instance? Could you explain how did you setup your EC2 instances, and so that we can try?
VPC CNI makes some fundamental assumptions about the underlying node requirements and VPC architecture. As far as I know, we may not have tried with a local tool like k3s or claimed support for it. The ipamd seems going a loop, am not sure if that is getting an IP address for allocation. Without CNI ready the Coredns wont come up too.

[IvanDimitrov2002]

Yes, I've deployed the k3s on two EC2 instances - one for the master (server in the context of k3s) and one worker (agent) node. For setting up the server node, I use the server node init command I provided above. For the agent node, its similar:

K3S_TOKEN=$(aws secretsmanager get-secret-value --secret-id k3s-token --query SecretString --output text)
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent \
  --kubelet-arg cloud-provider=external \
  --node-name '$(hostname -f)'" \
  K3S_URL="https://${aws_instance.k3s_server.private_ip}:6443" \
  K3S_TOKEN="$K3S_TOKEN" sh -s -

Where ${aws_instance.k3s_server.private_ip} is the private IP of the server node. (I'm using Terraform, if you wonder about the syntax, but I don't think its relevant). My agent node is controlled by an ASG but currently I've limited it to just 1 instance and disabled health checks for testing purposes. The K3S_TOKEN is kept as an aws secret and you can get it from /var/lib/rancher/k3s/server/node-token on the server node instance.

I set my KUBECONFIG environment like this:
echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" | sudo tee -a /etc/environment because K3S stores it on that location. In that kubeconfig file (k3s.yaml), k3s always names the cluster and the context as default. That being said, I have a couple of commands that change this naming to something like app-prod just so its more clear. K3S doesn't store info about the cluster naming so from its side its not a big deal. The reasons for renaming my cluster were:

1. Its more clear than default
2. I saw that for the CCM, I have to tag some of my resources with "kubernetes.io/cluster/${var.cluster_id}" = "owned" and so I did - the instances, the SGs and the subnets. (${var.cluster_id} = "app-prod" in that case)

After these commands, 2 HelmCharts are placed at /var/lib/rancher/k3s/server/manifests. They are executed by the k3s built-in Helm controller. Essentially, this process bootstraps the cluster by installing the AWS CCM and the VPC CNI:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: aws-cloud-controller-manager
  namespace: kube-system
spec:
  bootstrap: true
  chart: aws-cloud-controller-manager
  repo: https://kubernetes.github.io/cloud-provider-aws
  targetNamespace: kube-system
  valuesContent: |
    hostNetworking: true
    args:
      - --v=2
      - --cloud-provider=aws
      - --configure-cloud-routes=false
    nodeSelector:
      node-role.kubernetes.io/control-plane: "true"

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: aws-vpc-cni
  namespace: kube-system
spec:
  bootstrap: true
  chart: aws-vpc-cni
  repo: https://aws.github.io/eks-charts
  targetNamespace: kube-system
  valuesContent: |-
    enableNetworkPolicy: "false"
    image:
      override: public.ecr.aws/eks/amazon-k8s-cni:v1.16.4
    init:
      image:
        override: public.ecr.aws/eks/amazon-k8s-cni-init:v1.16.4
    nodeAgent:
      image:
        override: public.ecr.aws/eks/aws-network-policy-agent:v1.0.8
    env:
      WARM_IP_TARGET: 2
      MINIMUM_IP_TARGET: 10

As you can see, the only difference compared to the normal VPC CNI values.yaml is that I pull from the global public registry instead of the EKS specific regions (because I don't use EKS). I haven't even deployed the Network Policy controller yet.

So after the Helm controller installs the two helm charts, my nodes are ready and my pods in the kube-system namespaces are:

root@ip-10-0-1-205:/# kubectl get pods -n kube-system -o wide
NAME                                              READY   STATUS             RESTARTS          AGE   IP           NODE                                          NOMINATED NODE   READINESS GATES
helm-install-aws-cloud-controller-manager-crwmd   0/1     Completed          0                 27h   10.0.1.205   ip-10-0-1-205.eu-central-1.compute.internal   <none>           <none>
helm-install-aws-vpc-cni-cmnh2                    0/1     Completed          0                 27h   10.0.1.205   ip-10-0-1-205.eu-central-1.compute.internal   <none>           <none>
aws-node-672zj                                    2/2     Running            0                 27h   10.0.3.97    ip-10-0-3-97.eu-central-1.compute.internal    <none>           <none>
aws-cloud-controller-manager-k2p6w                1/1     Running            1 (27h ago)       27h   10.0.1.205   ip-10-0-1-205.eu-central-1.compute.internal   <none>           <none>
aws-node-q2h6l                                    2/2     Running            0                 27h   10.0.1.205   ip-10-0-1-205.eu-central-1.compute.internal   <none>           <none>
coredns-6799fbcd5-8svpr                           0/1     CrashLoopBackOff   409 (3m29s ago)   27h   10.0.3.96    ip-10-0-3-97.eu-central-1.compute.internal    <none>           <none>
metrics-server-67c658944b-7rh6d                   0/1     CrashLoopBackOff   303 (2m20s ago)   27h   10.0.1.238   ip-10-0-1-205.eu-central-1.compute.internal   <none>           <none>
local-path-provisioner-6c86858495-v2q25           0/1     CrashLoopBackOff   303 (108s ago)    27h   10.0.1.91    ip-10-0-1-205.eu-central-1.compute.internal   <none>           <none>

(a lot of restarts for the last 3 pods, I left them overnight)

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.