VK+KIP displays cells from a previous incarnation about kip

[justnoise]

Yes, this is actually expected (but not ideal). I'll go through what's happening to cause this and the way we deal with the issue in deploy/virtual-kubelet.yaml and how I deal with the issue when working with kip in minikube. This got a bit long, apologies.

Each Kip pod needs to keep track of which cloud instances it is responsible for (starting, stopping, cleaning up). To do this, each kip pod, when it first creates its internal state storage, also creates a UUID that identifies the provider. That UUID is the ControllerID.

Each cloud instance gets tagged with the ControllerID when it is created. The controller will only manipulate (start, stop, etc.) instances that have a ControllerID tag that matches their internal ControllerID. Likewise, since there might be multiple virtual-kubelet controllers in the same k8s cluster, each controller only maintains cell records that have a ControllerID field that matches their internal ControllerID value (the ControllerID field is stored in the Cels CRD but is not a default field for printing, you can see the field with kubectl get cells -oyaml).

So in the basic case, when you create a virtual kubelet pod, Kip will create its persistent storage and create a ControllerID which is written to that storage. Without patching kube-proxy, the controller will also launch a kube-proxy pod (tagged with the ControllerID) and create a cell for that pod. Note: We don't actually need/use the kube-proxy pod (I patch it out in my setup).

Now, when the virtual-kubelet pod is deleted, the instance and cell remain in the system marked with the original ControllerID. When the Kip virtual-kubelet deployment is re-created, the pod gets a new filesystem, the storage area for Kip is re-created and the controller creates a new ControllerID value. Kip will no longer track the old cells because it has a new identity.

How we fix this in deploy/virtual-kubelet.yaml:

1. In the manifests we give to users, we create a PersistentVolumeClaim for the virtual-kubelet storage and store Kip's state on that volume. When the pod is restarted or moved, the storage should remain the same.
2. In minikube, the situation is a bit different. I choose to store kip's data on a hostPath volume that's mounted into the pod. That remains the same across pod restarts but will go away if we do minikube delete:

  containers:
   - name: virtual-kubelet
     volumeMounts:
     - name: data
       mountPath: /opt/kip
  volumes:
  - name: data
    hostPath:
      path: /opt/kip
      type: DirectoryOrCreate

Both of these present a non-ideal situation but one that works and was chosen as one of the easiest ways to do it. When the time is right, I'd like to look into alternatives to preserving controller identity across pod restarts.

[ldx]

Created #131 to deal with all resources left behind by a deleted Kip instance.