Comments (2)
Yes, this is actually expected (but not ideal). I'll go through what's happening to cause this and the way we deal with the issue in deploy/virtual-kubelet.yaml
and how I deal with the issue when working with kip in minikube. This got a bit long, apologies.
Each Kip pod needs to keep track of which cloud instances it is responsible for (starting, stopping, cleaning up). To do this, each kip pod, when it first creates its internal state storage, also creates a UUID that identifies the provider. That UUID is the ControllerID
.
Each cloud instance gets tagged with the ControllerID
when it is created. The controller will only manipulate (start, stop, etc.) instances that have a ControllerID
tag that matches their internal ControllerID. Likewise, since there might be multiple virtual-kubelet controllers in the same k8s cluster, each controller only maintains cell records that have a ControllerID
field that matches their internal ControllerID
value (the ControllerID
field is stored in the Cels CRD but is not a default field for printing, you can see the field with kubectl get cells -oyaml
).
So in the basic case, when you create a virtual kubelet pod, Kip will create its persistent storage and create a ControllerID
which is written to that storage. Without patching kube-proxy, the controller will also launch a kube-proxy pod (tagged with the ControllerID
) and create a cell for that pod. Note: We don't actually need/use the kube-proxy pod (I patch it out in my setup).
Now, when the virtual-kubelet pod is deleted, the instance and cell remain in the system marked with the original ControllerID
. When the Kip virtual-kubelet deployment is re-created, the pod gets a new filesystem, the storage area for Kip is re-created and the controller creates a new ControllerID
value. Kip will no longer track the old cells because it has a new identity.
How we fix this in deploy/virtual-kubelet.yaml:
- In the manifests we give to users, we create a
PersistentVolumeClaim
for the virtual-kubelet storage and store Kip's state on that volume. When the pod is restarted or moved, the storage should remain the same. - In minikube, the situation is a bit different. I choose to store kip's data on a hostPath volume that's mounted into the pod. That remains the same across pod restarts but will go away if we do
minikube delete
:
containers:
- name: virtual-kubelet
volumeMounts:
- name: data
mountPath: /opt/kip
volumes:
- name: data
hostPath:
path: /opt/kip
type: DirectoryOrCreate
Both of these present a non-ideal situation but one that works and was chosen as one of the easiest ways to do it. When the time is right, I'd like to look into alternatives to preserving controller identity across pod restarts.
from kip.
Created #131 to deal with all resources left behind by a deleted Kip instance.
from kip.
Related Issues (20)
- Add map of instance families that support running as dedicated hosts to instance selector
- Provisioning documentation and configuration for standing up a cluster that can run mac1 instances
- Implement instance selector back-pressure
- Make sure dedicated host is "available" before launching an instance onto it
- Make sure the Base AMI is ready for the VM template HOT 1
- Set all needed EBS params in RunInstance request
- Redundant Ticker HOT 1
- Implement GetDNSInfo for azure
- Document provider configuration for azure
- Fix parsing virtualNetwork name for azure
- Create azure VM cell image HOT 1
- KIP init-cert does not work on EKS 1.19
- updating pricing data for EC2 instances stopped working
- KIP does not support BoundServiceAccount Token Volume
- Terraform main.tf doesn't work with mac with apple m1 chip
- Add smoke test for GKE
- service account token missing in pod HOT 1
- failing container does not increase Restart count, and log content is unexpected
- issues deploying pods to kip provider in minikube
- Singularity support in KIP provider
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kip.