Repository for GitOps Implementation with Kubeedge
The minimal tools required for creating a development environment for testing GitOps implementation in a k8s environment.
- Kubernetes Cluster, microK8s has been used for such purpouse.
- Helm, for installing required tools in the cluster not included in microk8s
- ArgoCD, as CD tool
Two development environments for creating the required Kubernetes Cluster has been used:
- macOs (amd64)
- Ubuntu 20.04 LTS (amd64)
The macOs installation includes instructions to install the tools required to create the cluster using microk8s and inteect with it (helm, kubectl).
Brew (the missing package manager) is required for installation.
Follow the instructions included in brew reference
Using brew
Installation using brew
Check brew status
brew info
kubectl
brew install kubernetes-cli
helm
brew install helm
References:
Install microk8s using homebrew
brew install ubuntu/microk8s/microk8s
Installed microk8s will create a VM for microk8s using multipass
To create a microk8s cluster (one node)
microk8s install --channel 1.21 --cpu --mem --disk
microk8s status --wait-ready
Once installed, verity installation
multipass list
multipass info microk8s-vm
Install required features:
microk8s enable
microk8s allows to execute kubctl commands in a very similar way
microk8s kubectl cluster-info
For developing purpouses is more common using the kubectl client. For this is omñy required to generate the config file.
microk8s config > $HOME/.kube/config
Checking configuration
kubectl cluster-info
Snap will be used for installation
sudo apt update
sudo apt install snapd
Use snap for installation
sudo snap install kubectl --classic
curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
References:
List available channels
sudo snap info microk8s
After checking compatibity with kubeedge version
sudo snap install microk8s --classic --channel=1.22/stable
Enable basic services
microk8s enable dns dashboard storage
Check status
microk8s status
For developing purpouses is more common using the kubectl client. So the following command is used to generate the reuired config file.
microk8s config > $HOME/.kube/config
Checking configuration
kubectl cluster-info
Add helm repo for argoCD
ArgoCD can be installed using helm. The values (values.yml) used for it are available in infra/argocd.
helm repo add argo https://argoproj.github.io/argo-helm
The values used for it are available in this repository in infra/charts/argocd
helm install argo-cd --create-namespace --namespace argo-cd --values values.yaml --version 4.3.1 argo/argo-cd --debug --dry-run
NOTES: In order to access the server UI the following options are avaialble:
-
Port fowarding
kubectl port-forward service/argo-cd-argocd-server -n argo-cd 18080:443
To check the result open the browser on http://localhost:18080 and accept the certificate
-
enable ingress in the values file
server.ingress.enabled
and either- Add the annotation for ssl passthrough: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/ingress.md#option-1-ssl-passthrough
- Add the
--insecure
flag toserver.extraArgs
in the values file and terminate SSL at your ingress: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/ingress.md#option-2-multiple-ingress-objects-and-hosts
After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by executing:
kubectl -n argo-cd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
Alternative
You can create yout own chart for managing argoCD installation as described in:
https://medium.com/devopsturkiye/self-managed-argo-cd-app-of-everything-a226eb100cf0 https://medium.com/@andrew.kaczynski/gitops-in-kubernetes-argo-cd-and-gitlab-ci-cd-5828c8eb34d6
The Kubernetes Dashboard allows a basic monitoring and managemente of the cluster.
Once enabled in microk8s.
microk8s enable dashboard
Several options for accessing the dashboard
- Using port fowarding
kubectl port-forward -n kube-system service/kubernetes-dashboard 8080:443
To obatin the access token
kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
kubectl -n kube-system describe secret $token
token=$(kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
Since Ubuntu and CentOs are described as the supported runtimes, it's required to install Ubuntu on edge nodes.
The edgeNodes are Raspberry pi 4 Model B. Other devices supporting Ubuntu can be used.
These instructions are for for ubuntu 20.04.04 LTS (arm64).
Create sd boot for rpi
https://www.raspberrypi.com/documentation/computers/getting-started.html#using-raspberry-pi-imager
Once installed is required to change some parameters of the OS kernel in rpis to allow kubeedge execution.
Enable remote ssh Check config
https://roboticsbackend.com/install-ubuntu-on-raspberry-pi-without-monitor/
Look for system-boot partition in Ubuntu SD Card
Configure wifi access
In network-config add content
version: 2
ethernets:
eth0:
dhcp4: true
optional: true
wifis:
wlan0:
dhcp4: true
optional: true
access-points:
"YOUR_WIFI_NAME":
password: "YOUR_WIFI_PASSWORD"
In /user-data check ssh is enabled
...
# On first boot, set the (default) ubuntu user's password to "ubuntu" and expire user passwords
chpasswd:
expire: true
list:
- ubuntu:ubuntu
# Enable password authentication with the SSH daemon
ssh_pwauth: true
...
Username/password: ubuntu/ubuntu
Enable cgroup
modify /boot/firmware/cmdline.txt
add cgroup_enable=memory cgroup_memory=1
The content should be something like
console=serial0,115200 console=tty1 root=PARTUUID=738a4d67-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
Upgrade OS
Get list of avilable updates
sudo apt update
Upgrade OS
sudo apt upgrade
After this reboot
Change hostname
Set edgeNodeName
sudo hostnamectl set-hostname <edgeNodeName>
Add new hostNmae to it
vi /etc/hosts
After this Reboot
Install docker
Using docker script
curl -sSL https://get.docker.com | sh
Add user to docker group
sudo usermod -aG docker ${USER}
Reboot
sudo reboot
Install kubeedge
Reference:
Install keadmin
In the example keadm v1.9.2.
On cloudside
For amd64 (Ubuntu 20)
wget https://github.com/kubeedge/kubeedge/releases/download/v1.9.2/keadm-v1.9.2-linux-amd64.tar.gz
On edgeside
For arm64 (Ubuntu 20)
wget https://github.com/kubeedge/kubeedge/releases/download/v1.9.2/keadm-v1.9.2-linux-arm64.tar.gz
tar xvf keadm-v1.9.2-linux-arm64.tar.gz
mv <extracted> /usr/local/bin/keadm
chmod +x /usr/local/bin/keadm
Install CloudCore
Cloudside with k8s cluster access
keadm init --kube-config ${HOME}/.kube/config --advertise-address "KUBEDGE_CLOUDCORE_ADDRESS"
Obtain token to enroll edge nodes
keadm gettoken --kube-config ${HOME}/.kube/config
Status
tail -f /var/log/kubeedge/cloudcore.log
Rebooting
pkill cloudcore
nohup cloudcore > cloudcore.log 2>&1 &
Restart cloudcore (Cloud side)
ps aux | grep cloudcore
kill -9 PID
<PATH>/cloudcore &
If registered as service
Create a file names cloudcore.service in
[Unit]
Description=cloudcore.service
[Service]
Type=simple
ExecStart=/usr/local/bin/cloudcore
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
Start service
systemctl start cloudcore
Enable automatic start updocker rm -f $(docker ps -a -q) of service
systemctl enable cloudcore
Check service status
systemctl --type=service | grep cloudcore
Install EdgeCore
Edgeside in nodes
Enroll
keadm join --cloudcore-ipport=KUBEDGE_CLOUDCORE_ADDRESS:10000 --edgenode-name=<NODE-NAME> --token=<TOKEN> --kubeedge-version=<>
Check status
journalctl -u edgecore.service -b
Restart edgecore (Edge side)
sudo systemctl restart edgecore
SWAGGER-API $ARGOCD_SERVER/swagger-ui
Obtain token
curl $ARGOCD_SERVER/api/v1/session -d $'{"username":"admin","password":"password"}'
Use API
curl $ARGOCD_SERVER/api/v1/applications -H "Authorization: Bearer $ARGOCD_TOKEN"
In order to enable metrics is required to make some actions.
Obtain K8S certificates
Generate certificates.
The cluster CA is required for enabling comunication between cloudcore and edgecore.
This are usually located in /etc/kubernetes/pki in microk8s these are located in /var/snap/microk8s/current/certs/
CA file and key
- K8SCA_FILE, /etc/kubernetes/pki/ca.crt
- K8SCA_KEY_FILE, /etc/kubernetes/pki/ca.key
In microk8s
- K8SCA_FILE, /var/snap/microk8s/current/certs/ca.crt
- K8SCA_KEY_FILE, /var/snap/microk8s/current/certs/ca.key
It's also required to declare the CLOUDCOREIPs.
Copy certgen.sh to /etc/kubeedge
To generate the certificates required for allowing remote loging
## Set working directory
cd /etc/kubeedge
# Declare vars
export CLOUDCOREIPS="[servers]"
export K8SCA_FILE=/var/snap/microk8s/current/certs/ca.crt
export K8SCA_KEY_FILE=/var/snap/microk8s/current/certs/ca.key
# Generate certificates
./certgen.sh stream
Once these certificates are generated it's required to modify the iptables and the keubedge configuration files on both cloudside and edgeside.
On cloudside
Modify /etc/kubeedge/config/cloudcore.yaml
Enable stream
cloudStream:
# change
enable: true
export CLOUDCOREIPS="[servers]"
iptables -t nat -A OUTPUT -p tcp --dport 10350 -j DNAT --to $CLOUDCOREIPS:10003
On edge side (every edge node)
Modify /etc/kubeedge/config/edgecore.yaml
Enable stream
edgeStream:
# change
enable: true
# Check value of cloudserver
server: [CLOUDCOREIP]:10004
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
Restart cloudside and edge side
# On cloudside
systemctl restart cloudcore
# On edgeside (every edge node)
systemctl restart edgecore
Once done, tt will be possible to access node statistics throw the common interfaces.
# On cloud sice
curl "http://[NODE_IP]:10351/stats/summary?only_cpu_and_memory=true"
# On edge sice
curl "http://localhost:10350/stats/summary?only_cpu_and_memory=true"
{
"node": {
"nodeName": "edgenode01",
"systemContainers": [
{
"name": "kubelet",
"startTime": "2022-10-22T16:17:01Z",
"cpu": {
"time": "2022-10-22T18:45:07Z",
"usageNanoCores": 36490670,
"usageCoreNanoSeconds": 252662549983
},
"memory": {
"time": "2022-10-22T18:45:07Z",
"usageBytes": 30613504,
"workingSetBytes": 30613504,
"rssBytes": 27066368,
"pageFaults": 22869,
"majorPageFaults": 0
}
},
{
"name": "runtime",
"startTime": "2022-10-20T16:47:00Z",
"cpu": {
"time": "2022-10-22T18:45:13Z",
"usageNanoCores": 13118760,
"usageCoreNanoSeconds": 1786750198925
},
"memory": {
"time": "2022-10-22T18:45:13Z",
"usageBytes": 115150848,
"workingSetBytes": 51351552,
"rssBytes": 36360192,
"pageFaults": 19054431,
"majorPageFaults": 297
}
},
{
"name": "pods",
"startTime": "2022-10-20T16:47:31Z",
"cpu": {
"time": "2022-10-22T18:45:03Z",
"usageNanoCores": 0,
"usageCoreNanoSeconds": 0
},
"memory": {
"time": "2022-10-22T18:45:03Z",
"availableBytes": 3977527296,
"usageBytes": 0,
"workingSetBytes": 0,
"rssBytes": 0,
"pageFaults": 0,
"majorPageFaults": 0
}
}
],
"startTime": "2022-10-20T16:47:01Z",
"cpu": {
"time": "2022-10-22T18:45:02Z",
"usageNanoCores": 62053022,
"usageCoreNanoSeconds": 9409379988290
},
"memory": {
"time": "2022-10-22T18:45:02Z",
"availableBytes": 2643247104,
"usageBytes": 2401181696,
"workingSetBytes": 1334280192,
"rssBytes": 165142528,
"pageFaults": 76527,
"majorPageFaults": 165
},
"network": {
"time": "2022-10-22T18:45:02Z",
"name": "eth0",
"rxBytes": 680926297,
"rxErrors": 0,
"txBytes": 43422651,
"txErrors": 0,
"interfaces": [
{
"name": "wlan0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 680926297,
"rxErrors": 0,
"txBytes": 43422651,
"txErrors": 0
}
]
},
"fs": {
"time": "2022-10-22T18:45:02Z",
"availableBytes": 22324531200,
"capacityBytes": 31064162304,
"usedBytes": 7425003520,
"inodesFree": 1805511,
"inodes": 1933312,
"inodesUsed": 127801
},
"runtime": {
"imageFs": {
"time": "2022-10-22T18:45:02Z",
"availableBytes": 22324531200,
"capacityBytes": 31064162304,
"usedBytes": 1395364102,
"inodesFree": 1805511,
"inodes": 1933312,
"inodesUsed": 127801
}
},
"rlimit": {
"time": "2022-10-22T18:45:17Z",
"maxpid": 4194304,
"curproc": 204
}
},
"pods": []
}
Since edgenodes are not really k8s nodes it's required to customize the metrics-server used for accessing node metrics.
In this case, version 5.2 of metrics server has been used.
wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.2/components.yaml -O deploy.yaml
And customized with the following values:
The file is avaliable in deploy.yaml
Enable hostnetwork mode
spec:
template:
spec:
hostNetwork: true
Skip tls insecure
spec:
template:
spec:
containers:
- args:
- --kubelet-insecure-tls
Node selector the service must be deployed in master node
spec:
template:
spec:
nodeSelector:
beta.kubernetes.io/arch : amd64
Once modified it can be applyed using
kubectl apply -f deploy.yaml
Once completed the following info should be available
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
edgenode01 48m 1% 1272Mi 34%
edgenode02 41m 1% 1169Mi 31%
rpi3 97m 2% 403Mi 49%
ubuntu-desktop 1688m 21% 9959Mi 41%
edgenode03 <unknown> <unknown> <unknown> <unknown>
Observing can be done using Grafana with Prometheus. Prometheus can be installed in several ways:
- Using included prometheus in microk8s distribution
- Unsing the helm chart provided by prometheus-community.
It will use Grafana and Prometheus included in microk8s distribution.
microsk8s enable prometheus
Once enabled these can be accesed through port fowarding
Set port-forwarding to enable external access
PrometheusUI
$ microk8s kubectl port-forward -n monitoring service/prometheus-k8s --address 0.0.0.0 9090:9090
Forwarding from 0.0.0.0:9090 -> 9090
Grafana UI
$ microk8s kubectl port-forward -n monitoring service/grafana --address 0.0.0.0 3000:3000
```txt
Forwarding from 0.0.0.0:3000 -> 3000
The metrics consumed in prometheus are configured to obtain data from port 10250.
The edgenodes metrics are avaliable on port 10350, so it's required to modify the endpoints used for retreaving node statistics.
Since modifing the default exporters included that is a hard job. It's recomended to use the Promehteus chart.
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Include and update chart in helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm show values prometheus-community/kube-prometheus-stack
Once modified the values to include the configuration required to accesss the metrics endpoints in edgenodes install the chart.
The selected chart must be compatible with kubernetes version.
helm install prometheus --create-namespace --namespace monitoring --values values.yaml --version [version] prometheus-community/kube-prometheus-stack --debug --dry-run
Check Promethues Service Monitor
kubectl describe service prometheus-operated -n monitoring
kubectl port-forward svc/kube-state-metrics 8080:8080
kubectl port-forward svc/prometheus-operated 9090:9090
kubectl port-forward svc/prometheus-grafana 3000:80
Show service monitors declared
kubectl get servicemonitor -monitoring
In order to allow access to containers running in edge nodes is required to install Kubeedge EdgeMesh.
Reference:
thisisunsafe