Comments (1)
Reproducer
Prerequisites
- Rancher 2.8.3 with a publicly-accessible Internet address (eg. in AKS)
- an Azure account
- its subscription id
- a resource group in that Azure account
Setup
- import any number of AKS clusters as managed clusters
- add Azure cloud credentials to Rancher (full docs). TL;DR:
- run:
export SUBSCRIPTION_ID=XXXXXXXX-YYYY-ZZZZ-AAAA-BBBBBBBB export GROUP=st-rg az ad sp create-for-rbac \ --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$GROUP \ --role Contributor
- note the JSON output
- go to ☰ -> Cluster Management -> Cloud Credentials -> Create -> Azure
- paste
appId
from the JSON output to Client ID,password
to Secret,$SUBSCRIPTION_ID
to Subscription ID
- run:
- import the clusters as as managed clusters ☰ -> Cluster Management -> Import Existing -> Azure AKS
- add Azure cloud credentials to Rancher (full docs). TL;DR:
Test (of base version)
In one terminal with kubeconfig pointing to the upstream cluster run:
while true
do
for cluster in $(kubectl get clusters.management.cattle.io --no-headers -o custom-columns=name:.metadata.name | grep -v local)
do
kubectl patch clusters.management.cattle.io ${cluster} --type='json' -p='[{"op": "replace", "path": "/status/serviceAccountTokenSecret", "value":""}]'
done
done
This script forces Rancher to continuously execute a core function of AKS cluster deployment (generateAndSetServiceAccount
). This emulates the reported environment where clusters are frequently deployed and redeployed without the necessity of really redeploying AKS clusters, which is time intensive.
In another terminal with kubeconfig pointing to the upstream cluster run:
while true; do
for pod in $(kubectl -n cattle-system get pods -l app=rancher --no-headers -o custom-columns=name:.metadata.name); do
kubectl exec -n cattle-system ${pod} -c rancher -- curl -s http://localhost:6060/debug/pprof/heap -o heap
kubectl cp -n cattle-system -c rancher ${pod}:heap ./heap
go tool pprof -top --trim=false --show_from=generateAndSetServiceAccount ./heap | grep kubernetes.NewForConfig
done
done
This script monitors one important source of memory usage by generateAndSetServiceAccount
(kubernetes.NewForConfig
).
Expected output is either:
- a continuous stream of
ShowFrom expression matched no samples
, which means no memory was allocated or ShowFrom expression matched no samples
interleaved with lines like the following:
0 0% 7.72% 7.51MB 5.79% k8s.io/client-go/kubernetes.NewForConfig
The fourth number indicates memory usage. It should increase only slightly, if at all, after several minutes. Importantly, stopping the first script in the other terminal should make usage go down to zero in some minutes.
Actual output is:
ShowFrom expression matched no samples
interleaved with lines like the following:
0 0% 7.72% 7.51MB 5.79% k8s.io/client-go/kubernetes.NewForConfig
The fourth number indicates memory usage. It is increasing, and stopping the first script in the other terminal does not make usage go down to zero.
Test (of patched version)
When the expected behavior above is reproduced:
- stop both scripts
- swap the Rancher image with the patched one:
kubectl set image -n cattle-system deployment/rancher rancher=rancher/rancher:v2.8.3-debug-45509-1
- wait for clusters to come back as fully available in the UI, refresh the homepage to double check that is the case
- re-run the scripts. Expected behavior should be observed
from rancher.
Related Issues (20)
- [BUG] rancher monitoring chart lacks Network Policy permission to collect metrics from GUI's ingress-nginx pods
- [BUG] cannot drop the username and password of a private registry in the secret cattle-system/cattle-private-registry in the downstream cluster once it is set on RKE1 downstream cluster
- [BUG] K3s worker only node does not work with system-default-agent args
- [BUG] cluster state stuck at "Waiting for etcd snapshot creation management plane restart probes"
- [BUG] Git charts repo validation does not allow hyphens in GitHub org names
- golang/mock is archived, switch to an actively maintained fork
- [BUG] UI - Project members not initially shown in project view
- [BUG] Re-added (removed and imported) RKE2 cluster fails to upgrade
- [BUG] Rancher Logging : "cluster.local" domain is used instead of actual domain of the cluster
- Cleanup PSP-related Resources from Rancher-Charts
- Clean up psp related resources that maybe present in the cluster after upgrade to k8s 1.29
- [BUG] External Access to Rancher Uses Internal Domain for Login, Causing Login Failure
- Jira Placeholder
- [RFE] PrincipalID refactor HOT 1
- add new agent-tls-mode setting
- add helm chart variable to set agent-tls-mode setting HOT 1
- CA check improvements for `fleet-agent` HOT 1
- Using install-docker script URL from settings.go HOT 1
- [BUG] etcd snapshots in Rancher UI shows incorrect age
- [EPIC] SQLite backed cache in the Steve API: feature parity
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rancher.