project-codeflare / multi-cluster-app-dispatcher Goto Github PK

View Code? Open in Web Editor NEW

106.0 13.0 62.0 73.48 MB

Holistic job manager on Kubernetes

License: Apache License 2.0

Makefile 1.40% Shell 16.47% Go 81.96% Dockerfile 0.08% Mustache 0.10%

fifo queueing resource-abstraction multi-cluster quota-management priority

multi-cluster-app-dispatcher's People

Contributors

Stargazers

Watchers

Forkers

dmatch01 purphury jenellemillison 0r0i zhangliresearch wangchen615 sonvt1710 bhaskers-blu-org1 zhaohc10 attlee-wang nhan-hoang tripathysa lynnsong qasim-at-tci hfwen0502 warmchang starpit dmitrigekhtman metalcycling kannon92 maxusmusti heyselbi asm582 mellis-github jbusche anishasthana z103cb kpostoffice kevin85421 mamy-cs lionelvillard sunya-ch lowang-bh vishakha-ramani srihari1192 dimakis jamindy maia-iyer fiona-waters codificat vanillaspoon bobbins228 sutaakar cybernetics christianzaccaria atantawi cwiklik revit13 eranra tedhtchang huangyingting ronensc mingchen-github sara-ks tardieu dgrove-oss elevo-ai menglingwei

multi-cluster-app-dispatcher's Issues

Remove default Helm image and related deployment documentation.

There is no Helm default docker registry image for opensource. Current default is invalid.

Controller crash while deleting job

It was observed that the controller crash while deleting the job, please find below the log snippet:

I0802 16:51:25.998003       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:26.007502       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:26.024477       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:26.308024       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:26.324680       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:26.756775       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:27.038065       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:27.202031       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:27.915954       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:27.995709       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:28.407129       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:28.791048       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:28.937964       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.057592       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.237928       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.445928       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.524904       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.729828       1 queuejob_controller_ex.go:1359] [UpdateQueueJobs] aw-helloworld *Delay=2036.739186 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc001946e00 Version=1811878 Status={Pending:0 Running:1 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Running ControllerFirstTimestamp:2022-08-02 16:17:32.990628 +0000 UTC FilterIgnore:true Sender:before [syncQueueJob] setRunning Local:false Conditions:[{Type:Init Status:True LastUpdateMicroTime:2022-08-02 16:17:32.990635 +0000 UTC LastTransitionMicroTime:2022-08-02 16:17:32.990637 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2022-08-02 16:17:32.990651 +0000 UTC LastTransitionMicroTime:2022-08-02 16:17:32.990651 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2022-08-02 16:17:32.998483 +0000 UTC LastTransitionMicroTime:2022-08-02 16:17:32.998485 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2022-08-02 16:17:33.071898 +0000 UTC LastTransitionMicroTime:2022-08-02 16:17:33.071898 +0000 UTC Reason:AppWrapperRunnable Message:} {Type:Running Status:True LastUpdateMicroTime:2022-08-02 16:17:39.384475 +0000 UTC LastTransitionMicroTime:2022-08-02 16:17:39.384475 +0000 UTC Reason:PodsRunning Message:}]}
I0802 16:51:29.730144       1 pod.go:250] [UpdateQueueJobStatus] There are 1 pods of QueueJob aw-helloworld:  pending 0, running 1, succeeded 0, failed 0
I0802 16:51:29.760911       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.874443       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:29.958165       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>
I0802 16:51:30.053859       1 cache.go:330] [deleteJob] Attempting to delete Job <7c070bc6-4afc-4c11-a881-81ee6e5e3b83:/>

This caused the worker thread to panic:

E0802 15:20:40.433541       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 445 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1c60da0, 0x2e48470)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x86
panic(0x1c60da0, 0x2e48470)
        /usr/local/go/src/runtime/panic.go:965 +0x1b9
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GetAPIVersion(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/unstructured.go:218
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GroupVersionKind(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/unstructured.go:426 +0x6c
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/genericresource.(*GenericResources).SyncQueueJob(0xc0007c10b0, 0xc000505700, 0xc0005b3728, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/genericresource/genericresource.go:205 +0x1a77
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).manageQueueJob(0xc000144b40, 0xc000505700, 0xc000052100, 0x0, 0x0)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1720 +0xa3c
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).syncQueueJob(0xc000144b40, 0xc000505700, 0x0, 0x0)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1590 +0x57f
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).worker.func1(0x1e95fc0, 0xc000505700, 0xc000f22b10, 0x21)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1518 +0x2ff
k8s.io/client-go/tools/cache.(*FIFO).Pop(0xc0002bdb00, 0xc0005b3e48, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/fifo.go:303 +0x1e6
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).worker(0xc000144b40)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1498 +0x59
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006bca50)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006bca50, 0x2178c80, 0xc000b78ed0, 0x10001, 0xc0000fe300)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006bca50, 0x0, 0x0, 0x1, 0xc0000fe300)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0006bca50, 0x0, 0xc0000fe300)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).Run
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1277 +0x885
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa11c0c]

goroutine 445 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109
panic(0x1c60da0, 0x2e48470)
        /usr/local/go/src/runtime/panic.go:965 +0x1b9
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GetAPIVersion(...)
        /go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/unstructured.go:218
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GroupVersionKind(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/unstructured/unstructured.go:426 +0x6c
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/genericresource.(*GenericResources).SyncQueueJob(0xc0007c10b0, 0xc000505700, 0xc0005b3728, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/genericresource/genericresource.go:205 +0x1a77
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).manageQueueJob(0xc000144b40, 0xc000505700, 0xc000052100, 0x0, 0x0)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1720 +0xa3c
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).syncQueueJob(0xc000144b40, 0xc000505700, 0x0, 0x0)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1590 +0x57f
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).worker.func1(0x1e95fc0, 0xc000505700, 0xc000f22b10, 0x21)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1518 +0x2ff
k8s.io/client-go/tools/cache.(*FIFO).Pop(0xc0002bdb00, 0xc0005b3e48, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/[email protected]/tools/cache/fifo.go:303 +0x1e6
github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).worker(0xc000144b40)
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1498 +0x59
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006bca50)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0006bca50, 0x2178c80, 0xc000b78ed0, 0x10001, 0xc0000fe300)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006bca50, 0x0, 0x0, 0x1, 0xc0000fe300)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0006bca50, 0x0, 0xc0000fe300)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob.(*XController).Run
        /go/src/github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:1277 +0x885

Pod and replicasets are not deleted after deleting the owning appwrapper

$ cat aw-deployment.0.yaml
apiVersion: arbitrator.incubator.k8s.io/v1alpha1
kind: AppWrapper
metadata:
name: hellodiana-aw-1-test-0
spec:
schedSpec:
minAvailable: 2
resources:
Items:
- replicas: 1
type: Deployment
template:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: hellodiana-aw-1-test-0
labels:
app: hellodiana-aw-1-test-0
spec:
selector:
matchLabels:
app: hellodiana-aw-1-test-0
replicas: 1
template:
metadata:
labels:
app: hellodiana-aw-1-test-0
spec:
containers:
- name: hellodiana-aw-1-test-0
#image: gcr.io/hello-minikube-zero-install/hello-node
image: k8s.gcr.io/echoserver:1.4
resources:
limits:
cpu: 125m
memory: 1024Mi
requests:
cpu: 125m
memory: 1024Mi

$ k get appwrappers; k get deployments; k get rs; k get pods

No resources found.
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
failfast-operator 1 1 1 1 281d
NAME DESIRED CURRENT READY AGE
failfast-operator-f495b4f99 1 1 1 281d
NAME READY STATUS RESTARTS AGE
failfast-operator-f495b4f99-2rw2t 1/1 Running 1 281d

$ k create -f aw-deployment.0.yaml
$ k get appwrappers; k get deployments; k get rs; k get pods

NAME AGE
hellodiana-aw-1-test-0 19s
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
failfast-operator 1 1 1 1 281d
NAME DESIRED CURRENT READY AGE
failfast-operator-f495b4f99 1 1 1 281d
hellodiana-aw-1-test-0-57c949794 1 1 1 19s
NAME READY STATUS RESTARTS AGE
failfast-operator-f495b4f99-2rw2t 1/1 Running 1 281d
hellodiana-aw-1-test-0-57c949794-hckpk 1/1 Running 0 19s

$ k delete -f aw-deployment.0.yaml
$ k get appwrappers; k get deployments; k get rs; k get pods
No resources found.
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
failfast-operator 1 1 1 1 281d
NAME DESIRED CURRENT READY AGE
failfast-operator-f495b4f99 1 1 1 281d
hellodiana-aw-1-test-0-57c949794 1 1 1 1m
NAME READY STATUS RESTARTS AGE
failfast-operator-f495b4f99-2rw2t 1/1 Running 1 281d
hellodiana-aw-1-test-0-57c949794-hckpk 1/1 Running 0 1m

REGRESSION: After the delete is issued for the appwrapper, the replicaset and pod do not get deleted.

Pods not starting

Hi,
It seems like the MCAD-controller is just the tool I have been looking for, wanting to throttle some 1000+ jobs (each with ca 10min calculations) submitted onto our on-prem Kubernetes cluster.

I managed to deploy controler and it seem to be running OK, but jobs doesn't seem to get started.

I deployed with helm from master: helm install mcad-controller --namespace kube-system --wait --set resources.requests.cpu=1000m --set resources.requests.memory=1024Mi --set resources.limits.cpu=1000m --set resources.limits.memory=1024Mi --set image.repository=darroyo/mcad-controller --set image.tag=dmatch01-multi-cluster-agent-fixes-v1.29.6 --set image.pullPolicy=Always

I apply the example appwrappers from example: kubectl apply -f aw-01.yaml

I haver plenty of available resources on my cluster.

No pods get started:

kubectl get po
No resources found in default namespace.

Here are the appwrapper details:

kubectl get appwrappers.mcad.ibm.com
NAME                      AGE
stateful-set-2-replicas   3m57s

And in detail:

kubectl describe appwrappers.mcad.ibm.com stateful-set-2-replicas
Name:         stateful-set-2-replicas
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"mcad.ibm.com/v1alpha1","kind":"AppWrapper","metadata":{"annotations":{},"name":"stateful-set-2-replicas","namespace":"defau...
API Version:  mcad.ibm.com/v1alpha1
Kind:         AppWrapper
Metadata:
  Creation Timestamp:  2021-02-01T06:31:50Z
  Generation:          5
  Managed Fields:
    API Version:  mcad.ibm.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:resources:
          f:GenericItems:
          f:Items:
          f:metadata:
        f:schedulingSpec:
        f:service:
          .:
          f:spec:
      f:status:
        .:
        f:canrun:
        f:conditions:
        f:controllerfirsttimestamp:
        f:filterignore:
        f:queuejobstate:
        f:sender:
        f:state:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2021-02-01T06:31:50Z
    API Version:  mcad.ibm.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:resources:
    Manager:         kubectl
    Operation:       Update
    Time:            2021-02-01T06:31:50Z
  Resource Version:  335269363
  Self Link:         /apis/mcad.ibm.com/v1alpha1/namespaces/default/appwrappers/stateful-set-2-replicas
  UID:               9faf54aa-ecea-4d56-bf7a-3f22d6295ee0
Spec:
  Resources:
    Generic Items:  <nil>
    Items:
      Allocatedreplicas:  1
      Metadata:
        Creation Timestamp:  <nil>
        Labels:
          Qjob Res Label:  0
      Priority:            0
      Priorityslope:       0
      Replicas:            1
      Template:
        API Version:  apps/v1
        Kind:         StatefulSet
        Metadata:
          Labels:
            App:  stateful-set-2-replicas
          Name:   stateful-set-2-replicas
        Spec:
          Replicas:  2
          Selector:
            Match Labels:
              App:  stateful-set-2-replicas
          Template:
            Metadata:
              Labels:
                App:   stateful-set-2-replicas
                Size:  2
            Spec:
              Containers:
                Image:  k8s.gcr.io/echoserver:1.4
                Name:   stateful-set-2-replicas
                Resources:
                  Limits:
                    Memory:  200Mi
                  Requests:
                    Memory:  200Mi
      Type:                  StatefulSet
    Metadata:
  Scheduling Spec:
  Service:
    Spec:
Status:
  Canrun:  true
  Conditions:
    Last Transition Micro Time:  2021-02-01T06:31:50.462866Z
    Last Update Micro Time:      2021-02-01T06:31:50.462866Z
    Status:                      True
    Type:                        Init
    Last Transition Micro Time:  2021-02-01T06:31:50.474465Z
    Last Update Micro Time:      2021-02-01T06:31:50.474465Z
    Reason:                      AwaitingHeadOfLine
    Status:                      True
    Type:                        Queueing
    Last Transition Micro Time:  2021-02-01T06:31:50.494659Z
    Last Update Micro Time:      2021-02-01T06:31:50.494659Z
    Reason:                      FrontOfQueue.
    Status:                      True
    Type:                        HeadOfLine
    Last Transition Micro Time:  2021-02-01T06:31:50.630181Z
    Last Update Micro Time:      2021-02-01T06:31:50.630181Z
    Reason:                      AppWrapperRunnable
    Status:                      True
    Type:                        Dispatched
  Controllerfirsttimestamp:      2021-02-01T06:31:50.462865Z
  Filterignore:                  true
  Queuejobstate:                 Dispatched
  Sender:                        before [manageQueueJob]afterEtcdDispatching
  State:                         Running
Events:                          <none>

The logs from mcad-kontroller: kubectl logs -n kube-system mcad-controller:

kubectl logs -n kube-system mcad-controller-75d985d86c-bd2gb
I0201 06:28:01.353595       1 flags.go:52] FLAG: --agentconfigs=""
I0201 06:28:01.353633       1 flags.go:52] FLAG: --alsologtostderr="false"
I0201 06:28:01.353636       1 flags.go:52] FLAG: --backofftime="20"
I0201 06:28:01.353639       1 flags.go:52] FLAG: --dispatcher="false"
I0201 06:28:01.353643       1 flags.go:52] FLAG: --dynamicpriority="false"
I0201 06:28:01.353646       1 flags.go:52] FLAG: --headoflineholdingtime="0"
I0201 06:28:01.353663       1 flags.go:52] FLAG: --kubeconfig=""
I0201 06:28:01.353665       1 flags.go:52] FLAG: --log-backtrace-at=":0"
I0201 06:28:01.353670       1 flags.go:52] FLAG: --log-dir=""
I0201 06:28:01.353672       1 flags.go:52] FLAG: --log-flush-frequency="5s"
I0201 06:28:01.353676       1 flags.go:52] FLAG: --logtostderr="true"
I0201 06:28:01.353678       1 flags.go:52] FLAG: --master=""
I0201 06:28:01.353680       1 flags.go:52] FLAG: --preemption="false"
I0201 06:28:01.353683       1 flags.go:52] FLAG: --scheduler=""
I0201 06:28:01.353685       1 flags.go:52] FLAG: --stderrthreshold="2"
I0201 06:28:01.353687       1 flags.go:52] FLAG: --v="4"
I0201 06:28:01.353689       1 flags.go:52] FLAG: --vmodule=""
I0201 06:28:01.354815       1 builder.go:132] Entered Flags()
I0201 06:28:01.354823       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.354826       1 builder.go:115] initFlagSet(): FlagSet was nil.
I0201 06:28:01.354829       1 builder.go:120] initFlagSet(): Flagset secure-port = 0
I0201 06:28:01.354832       1 builder.go:88] Entered InstallFlags()
I0201 06:28:01.354834       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.354837       1 builder.go:120] initFlagSet(): Flagset secure-port = 0
I0201 06:28:01.355177       1 adapter.go:68]
I0201 06:28:01.355244       1 builder.go:132] Entered Flags()
I0201 06:28:01.355246       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.355249       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.355253       1 builder.go:124] initFlagSet(): Flagset secure-port = 443
I0201 06:28:01.355256       1 builder.go:88] Entered InstallFlags()
I0201 06:28:01.355259       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.355301       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.355305       1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0201 06:28:01.355312       1 builder.go:132] Entered Flags()
I0201 06:28:01.355314       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.355318       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.355321       1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0201 06:28:01.355323       1 builder.go:88] Entered InstallFlags()
I0201 06:28:01.355330       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.355332       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.355334       1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0201 06:28:01.355337       1 builder.go:132] Entered Flags()
I0201 06:28:01.355339       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.355341       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.355343       1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0201 06:28:01.355345       1 builder.go:88] Entered InstallFlags()
I0201 06:28:01.355348       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.355352       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.355355       1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0201 06:28:01.355359       1 adapter.go:49] Entered makeProviderOrDie()
I0201 06:28:01.518200       1 adapter.go:78] starting adapter...
I0201 06:28:01.518581       1 builder.go:292] Entered Run()
I0201 06:28:01.518598       1 builder.go:88] Entered InstallFlags()
I0201 06:28:01.518608       1 builder.go:111] Entered initFlagSet()
I0201 06:28:01.518623       1 builder.go:122] initFlagSet(): Error getting secure-port: <nil>
I0201 06:28:01.518636       1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0201 06:28:01.522202       1 queuejobresources.go:61] Registered queue job resource type Pod
I0201 06:28:01.522232       1 queuejobresources.go:61] Registered queue job resource type Service
I0201 06:28:01.522244       1 queuejobresources.go:61] Registered queue job resource type Deployment
I0201 06:28:01.522301       1 queuejobresources.go:61] Registered queue job resource type StatefulSet
I0201 06:28:01.522313       1 queuejobresources.go:61] Registered queue job resource type PersistentVolume
I0201 06:28:01.522486       1 queuejobresources.go:61] Registered queue job resource type PersistentVolumeClaim
I0201 06:28:01.522498       1 queuejobresources.go:61] Registered queue job resource type Namespace
I0201 06:28:01.522509       1 queuejobresources.go:61] Registered queue job resource type ConfigMap
I0201 06:28:01.522559       1 queuejobresources.go:61] Registered queue job resource type Secret
I0201 06:28:01.522579       1 queuejobresources.go:61] Registered queue job resource type NetworkPolicy
I0201 06:28:01.538280       1 queuejob_controller_ex.go:361] [Controller] Agent mode
E0201 06:28:01.538296       1 queuejobagent.go:62] [agentEventQueue] Invalid agent configuration: .  Agent cluster will not be instantiated.
I0201 06:28:01.541215       1 reflector.go:122] Starting reflector *v1.ConfigMap (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/configmap/configmap.go:112
I0201 06:28:01.541247       1 reflector.go:160] Listing and watching *v1.ConfigMap from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/configmap/configmap.go:112
I0201 06:28:01.541225       1 reflector.go:122] Starting reflector *v1.Pod (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/pod/pod.go:139
I0201 06:28:01.541403       1 reflector.go:160] Listing and watching *v1.Pod from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/pod/pod.go:139
I0201 06:28:01.543105       1 reflector.go:122] Starting reflector *v1.Secret (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/secret/secret.go:111
I0201 06:28:01.543144       1 reflector.go:160] Listing and watching *v1.Secret from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/secret/secret.go:111
I0201 06:28:01.543517       1 reflector.go:122] Starting reflector *v1.Namespace (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/namespace/namespace.go:111
I0201 06:28:01.544346       1 reflector.go:122] Starting reflector *v1.Service (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/service/service.go:111
I0201 06:28:01.544409       1 reflector.go:160] Listing and watching *v1.Service from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/service/service.go:111
I0201 06:28:01.544276       1 reflector.go:160] Listing and watching *v1.Namespace from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/namespace/namespace.go:111
I0201 06:28:01.543587       1 reflector.go:122] Starting reflector *v1.Deployment (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/deployment/deployment.go:179
I0201 06:28:01.544508       1 reflector.go:122] Starting reflector *v1.NetworkPolicy (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/networkpolicy/networkpolicy.go:113
I0201 06:28:01.544531       1 reflector.go:122] Starting reflector *v1alpha1.AppWrapper (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:828
I0201 06:28:01.544544       1 reflector.go:160] Listing and watching *v1.NetworkPolicy from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/networkpolicy/networkpolicy.go:113
I0201 06:28:01.544569       1 reflector.go:160] Listing and watching *v1alpha1.AppWrapper from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejob/queuejob_controller_ex.go:828
I0201 06:28:01.544534       1 reflector.go:160] Listing and watching *v1.Deployment from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/deployment/deployment.go:179
I0201 06:28:01.544312       1 reflector.go:122] Starting reflector *v1.PersistentVolumeClaim (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/persistentvolumeclaim/persistentvolumeclaim.go:110
I0201 06:28:01.544735       1 reflector.go:160] Listing and watching *v1.PersistentVolumeClaim from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/persistentvolumeclaim/persistentvolumeclaim.go:110
I0201 06:28:01.544396       1 reflector.go:122] Starting reflector *v1.PersistentVolume (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/persistentvolume/persistentvolume.go:111
I0201 06:28:01.544791       1 reflector.go:160] Listing and watching *v1.PersistentVolume from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/persistentvolume/persistentvolume.go:111
I0201 06:28:01.636879       1 reflector.go:122] Starting reflector *v1.StatefulSet (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/statefulset/statefulset.go:113
I0201 06:28:01.636921       1 reflector.go:160] Listing and watching *v1.StatefulSet from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources/statefulset/statefulset.go:113
I0201 06:28:01.841239       1 shared_informer.go:123] caches populated
I0201 06:28:01.842082       1 reflector.go:122] Starting reflector *v1alpha1.SchedulingSpec (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/clusterstate/cache/cache.go:167
I0201 06:28:01.842142       1 reflector.go:160] Listing and watching *v1alpha1.SchedulingSpec from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/clusterstate/cache/cache.go:167
I0201 06:28:01.842130       1 reflector.go:122] Starting reflector *v1.Node (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/clusterstate/cache/cache.go:166
I0201 06:28:01.842234       1 reflector.go:160] Listing and watching *v1.Node from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/clusterstate/cache/cache.go:166
I0201 06:28:01.842053       1 reflector.go:122] Starting reflector *v1.Pod (0s) from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/clusterstate/cache/cache.go:165
I0201 06:28:01.842386       1 reflector.go:160] Listing and watching *v1.Pod from github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/clusterstate/cache/cache.go:165
I0201 06:28:02.336500       1 event_handlers.go:199] [AddPod] Added pod <serving/dpaas-mariadb-6b74df5479-vcjqd> into cache.
......
......
......
I0201 06:28:02.342703       1 event_handlers.go:199] [AddPod] Added pod <monitoring/prometheus-node-exporter-tw7ds> into cache.
I0201 06:28:02.342727       1 event_handlers.go:199] [AddPod] Added pod <kube-system/coredns-849545576b-pwcjw> into cache.
I0201 06:28:03.256580       1 serving.go:293] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0201 06:28:03.612041       1 cmhandlers.go:39] Entered CMHandlers registerResourceHandlers()
I0201 06:28:03.612080       1 cmhandlers.go:80] CMHandlers registerResourceHandlers(): rootScopedPath={resource}/{name}/{subresource}
I0201 06:28:03.612095       1 cmhandlers.go:91] CMHandlers registerResourceHandlers(): namespacedPath=namespaces/{namespace}/{resource}/{name}/{subresource}
I0201 06:28:03.612103       1 cmhandlers.go:94] CMHandlers registerResourceHandlers(): namespaceSpecificPath=namespaces/{namespace}/metrics/{name}
I0201 06:28:03.612116       1 installer.go:287] Entered restfulListResource()
I0201 06:28:03.612129       1 installer.go:288] restfulListResource(): restLister=&{0xc0002f0070}, scope={{{{} true /apis/custom.metrics.k8s.io/v1beta1/ }} {0xc0002f1420 [{[application/json] application/json [json] true 0xc0005bc900 0xc0005bc940 []  {} 0xc0005bc900} {[application/yaml] application/yaml [yaml] true 0xc0005bc980 <nil> []  <nil> <nil>} {[application/vnd.kubernetes.protobuf] application/vnd.kubernetes.protobuf [pb] false 0xc00024c6e0 <nil> []  {} 0xc000674c60}] 0xc00083f7e0 [{application/json true 0xc0005bc900 0xc0005bc940 0xc000674cc0} {application/yaml true 0xc0005bc980 <nil> <nil>} {application/vnd.kubernetes.protobuf false 0xc00024c6e0 <nil> 0xc000674cf0}] 0xc0005bc900} 0xc000038880 0xc0002f1420 0xc0002f1420 <nil> 0xc0002f1420 {0xc0002f1420} <nil> <nil> <nil> {custom.metrics.k8s.io v1beta1 *} custom.metrics.k8s.io/v1beta1, Kind=MetricValue * meta.k8s.io/v1 0}
I0201 06:28:03.612327       1 installer.go:287] Entered restfulListResource()
I0201 06:28:03.612336       1 installer.go:288] restfulListResource(): restLister=&{0xc0002f0070}, scope={{{{} false /apis/custom.metrics.k8s.io/v1beta1/namespaces/ }} {0xc0002f1420 [{[application/json] application/json [json] true 0xc0005bc900 0xc0005bc940 []  {} 0xc0005bc900} {[application/yaml] application/yaml [yaml] true 0xc0005bc980 <nil> []  <nil> <nil>} {[application/vnd.kubernetes.protobuf] application/vnd.kubernetes.protobuf [pb] false 0xc00024c6e0 <nil> []  {} 0xc000674c60}] 0xc00083f7e0 [{application/json true 0xc0005bc900 0xc0005bc940 0xc000674cc0} {application/yaml true 0xc0005bc980 <nil> <nil>} {application/vnd.kubernetes.protobuf false 0xc00024c6e0 <nil> 0xc000674cf0}] 0xc0005bc900} 0xc000038880 0xc0002f1420 0xc0002f1420 <nil> 0xc0002f1420 {0xc0002f1420} <nil> <nil> <nil> {custom.metrics.k8s.io v1beta1 *} custom.metrics.k8s.io/v1beta1, Kind=MetricValue * meta.k8s.io/v1 0}
I0201 06:28:03.612430       1 installer.go:287] Entered restfulListResource()
I0201 06:28:03.612440       1 installer.go:288] restfulListResource(): restLister=&{0xc0002f0070}, scope={{{{} false /apis/custom.metrics.k8s.io/v1beta1/namespaces/ }} {0xc0002f1420 [{[application/json] application/json [json] true 0xc0005bc900 0xc0005bc940 []  {} 0xc0005bc900} {[application/yaml] application/yaml [yaml] true 0xc0005bc980 <nil> []  <nil> <nil>} {[application/vnd.kubernetes.protobuf] application/vnd.kubernetes.protobuf [pb] false 0xc00024c6e0 <nil> []  {} 0xc000674c60}] 0xc00083f7e0 [{application/json true 0xc0005bc900 0xc0005bc940 0xc000674cc0} {application/yaml true 0xc0005bc980 <nil> <nil>} {application/vnd.kubernetes.protobuf false 0xc00024c6e0 <nil> 0xc000674cf0}] 0xc0005bc900} 0xc000038880 0xc0002f1420 0xc0002f1420 <nil> 0xc0002f1420 {0xc0002f1420} <nil> <nil> <nil> {custom.metrics.k8s.io v1beta1 *} custom.metrics.k8s.io/v1beta1, Kind=MetricValue * meta.k8s.io/v1 0}
I0201 06:28:03.612706       1 emhandlers.go:38] Entered EMHandlers registerResourceHandlers()
I0201 06:28:03.612732       1 emhandlers.go:77] EMHandlers registerResourceHandlers(): externalMetricPath=namespaces/{namespace}/{resource}
I0201 06:28:03.612745       1 installer.go:287] Entered restfulListResource()
I0201 06:28:03.612752       1 installer.go:288] restfulListResource(): restLister=&{0xc0002f0070}, scope={{{{} false /apis/external.metrics.k8s.io/v1beta1/namespaces/ }} {0xc0002f1420 [{[application/json] application/json [json] true 0xc0005bc900 0xc0005bc940 []  {} 0xc0005bc900} {[application/yaml] application/yaml [yaml] true 0xc0005bc980 <nil> []  <nil> <nil>} {[application/vnd.kubernetes.protobuf] application/vnd.kubernetes.protobuf [pb] false 0xc00024c6e0 <nil> []  {} 0xc000674c60}] 0xc00083f7e0 [{application/json true 0xc0005bc900 0xc0005bc940 0xc000674cc0} {application/yaml true 0xc0005bc980 <nil> <nil>} {application/vnd.kubernetes.protobuf false 0xc00024c6e0 <nil> 0xc000674cf0}] 0xc0005bc900} 0xc000038880 0xc0002f1420 0xc0002f1420 <nil> 0xc0002f1420 {0xc0002f1420} <nil> <nil> <nil> {external.metrics.k8s.io v1beta1 *} external.metrics.k8s.io/v1beta1, Kind=ExternalMetricValue * meta.k8s.io/v1 0}
I0201 06:28:03.612931       1 builder.go:299] Successfully created server.
I0201 06:28:03.613191       1 secure_serving.go:116] Serving securely on [::]:6443
I0201 06:28:03.639240       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (789.794µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.639297       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (309.988µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.639309       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (286.568µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639351       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (252.518µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639509       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (787.164µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.639544       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (222.718µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639566       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (828.704µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639628       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (832.705µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.639671       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (1.490219ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.639710       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (168.709µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639533       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (140.969µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639860       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.544679ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.639875       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (939.974µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639922       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (185.589µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.639937       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (208.919µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.640187       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.832088ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.640247       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (297.668µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.640345       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (146.93µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.640508       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (356.068µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.640190       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (149.839µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.738106       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (190.838µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.738131       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (541.406µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.742535       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (103.986087ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.742938       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (103.693779ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.743426       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (5.871499ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.743440       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (523.206µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.744036       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.886036ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.837204       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (198.160612ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.837248       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (99.426369ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.837449       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (93.436481ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.837613       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (198.905338ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.837968       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (95.226488ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.838098       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (101.562894ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.838339       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (994.893µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.838761       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.943927ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.936631       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (98.466796ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.936814       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (99.226621ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.936960       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (99.242171ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.936972       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (193.10688ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.936998       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (99.707168ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.937052       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (193.925174ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.937113       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (99.246931ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.937311       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (100.82495ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.937546       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (98.659755ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.937568       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (756.585µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:03.937567       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (101.293857ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.937595       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (101.147168ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.937654       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (99.655329ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.937757       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (101.494075ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.937790       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (99.39001ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.937934       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (360.007µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.937970       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (99.182392ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.938016       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (99.41105ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.938390       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (102.103111ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.938414       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.137422ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.938416       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.262931ms) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:03.938477       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (101.987362ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.938514       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (102.28256ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.938546       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (102.090491ms) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:03.938670       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (941.073µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:04.304338       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (259.569µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.304340       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (237.779µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.304440       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (261.538µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.304442       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (313.028µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.304450       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (295.388µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.310845       1 wrap.go:47] GET /openapi/v2: (6.682633ms) 404 [ 10.42.3.0:55048]
I0201 06:28:04.315906       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (289.338µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.316127       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (300.718µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.316291       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (374.637µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.316359       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (298.548µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.316465       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (257.488µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:04.317185       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (1.888377ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:04.346860       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (244.869µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:04.392752       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (299.368µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.392794       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (325.048µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.392802       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (209.619µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.393015       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (274.938µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.393058       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (238.708µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.394747       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (247.508µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.395182       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (309.138µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.395342       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (321.388µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.395635       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (212.949µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:04.395653       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (275.058µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:04.396050       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (424.277µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:05.499057       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (316.258µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:05.518896       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (328.018µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:05.540312       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (260.208µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:06.107578       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (324.518µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107600       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (324.738µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107666       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (292.078µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107785       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (171.398µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107857       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (343.338µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107861       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (218.688µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107882       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (279.898µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.107973       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (256.638µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.108148       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (278.138µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.108166       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (243.368µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:06.525589       1 wrap.go:47] GET /openapi/v2: (323.828µs) 404 [ 10.42.0.0:1995]
I0201 06:28:06.527283       1 wrap.go:47] GET /openapi/v2: (321.678µs) 404 [ 10.42.0.0:1995]
I0201 06:28:06.842449       1 wrap.go:47] GET /openapi/v2: (250.818µs) 404 [ 10.42.2.0:31631]
I0201 06:28:06.843738       1 wrap.go:47] GET /openapi/v2: (315.928µs) 404 [ 10.42.2.0:31631]
I0201 06:28:11.094923       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (363.798µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:11.116254       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (450.597µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:11.137159       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (272.878µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:28:12.224910       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (317.167µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:12.291094       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (507.197µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:12.309802       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (278.348µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:28:12.708787       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.883862ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:28:13.007816       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.970479ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:28:13.606846       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (279.998µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.607551       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (128.269µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.607670       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (132.169µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.607926       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (126.929µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.608039       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (323.928µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.608604       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (190.478µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.608972       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (124.499µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.609165       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (118.579µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.609360       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (224.928µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:13.609638       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (188.408µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:14.630304       1 wrap.go:47] GET /openapi/v2: (3.189488ms) 404 [ 10.42.0.0:1995]
I0201 06:28:14.631835       1 wrap.go:47] GET /openapi/v2: (435.407µs) 404 [ 10.42.0.0:1995]
I0201 06:28:15.314171       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.582514ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:28:15.362529       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.528022ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:28:17.907404       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.417646ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:17.993119       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (474.736µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:28:18.051807       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (325.047µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:18.087703       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (316.888µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.088008       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (488.116µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.088039       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (335.738µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.088229       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (322.917µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.088369       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (220.648µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.088418       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (211.289µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.089171       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (232.999µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.089193       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (697.795µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.089343       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (248.769µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:18.089404       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (197.888µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:19.108706       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (337.558µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:19.130804       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (478.617µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:19.149225       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (297.558µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:24.907582       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (404.877µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:24.949539       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (458.727µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:24.990904       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (308.968µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:26.106427       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (497.776µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:26.129609       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (485.527µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:26.196263       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (270.208µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:28.154110       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.845613ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:28:28.203418       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.048659ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:28:32.015253       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.138488ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:32.038268       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (369.187µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:32.112135       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (312.428µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:33.198324       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (323.677µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:33.218533       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (336.037µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:33.240483       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (496.867µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:36.108405       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (336.688µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.108419       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (366.627µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.108559       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (192.599µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.108986       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (547.596µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.109070       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (486.747µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.109356       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (346.008µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.109377       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (222.609µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.109396       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (379.998µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.109724       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (425.558µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:36.109860       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (267.639µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:28:38.920754       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (393.787µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:38.995075       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (323.678µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:28:39.023995       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (324.568µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:40.201992       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (315.948µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:40.226086       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (267.119µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:40.292569       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (281.858µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:28:43.611422       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (416.527µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.611488       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (464.026µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.611638       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (482.876µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.611671       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (282.338µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.611774       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (488.586µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.611792       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (259.158µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.612020       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (396.437µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.612128       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (228.678µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.612227       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (183.879µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:43.612353       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (414.557µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:28:46.009571       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.948762ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:46.030023       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (382.148µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:46.051942       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (318.578µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:46.880188       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.358546ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:28:46.929546       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.553302ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:28:47.207817       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (344.007µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:47.235790       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (350.257µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:47.290352       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (301.808µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:48.090425       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (345.618µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.090794       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (573.076µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.091201       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (443.627µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.091484       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (374.907µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.092099       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (462.777µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.092172       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (353.597µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.093235       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (253.439µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.094266       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (216.389µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.094835       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (277.038µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:48.095718       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (246.238µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:28:49.292902       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.020899ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:28:49.345091       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (6.202796ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:28:53.115669       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (410.847µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:53.190868       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (340.297µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:53.208577       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (313.158µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:54.330826       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (329.778µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:54.351921       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (338.827µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:54.370439       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (420.677µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:28:59.370278       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.148931ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:28:59.420145       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.375936ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:29:00.116041       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.295837ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:00.156123       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (297.088µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:00.192284       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (543.426µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:01.298446       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (341.507µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:01.321002       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (341.018µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:01.393699       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (335.307µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:04.291605       1 wrap.go:47] GET /openapi/v2: (3.458086ms) 404 [ 10.42.3.0:55048]
I0201 06:29:04.312412       1 wrap.go:47] GET /openapi/v2: (173.558µs) 404 [ 10.42.3.0:55048]
I0201 06:29:04.546314       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:04.546350       1 event_handlers.go:225] [UpdatePod] Updated pod <haste/pipeline-client-7c8cc9d588-n78dk> in cache.
I0201 06:29:05.605072       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:05.605099       1 event_handlers.go:225] [UpdatePod] Updated pod <haste/pipeline-client-7c8cc9d588-n78dk> in cache.
I0201 06:29:06.109436       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.266381ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.109897       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.556829ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.110153       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (1.788927ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.110280       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (834.134µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.109899       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (1.780127ms) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.110659       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (621.246µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.111059       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (569.406µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.111181       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (472.046µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.111294       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (417.557µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.111844       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (759.664µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:06.844452       1 wrap.go:47] GET /openapi/v2: (423.166µs) 404 [ 10.42.2.0:31631]
I0201 06:29:06.845737       1 wrap.go:47] GET /openapi/v2: (364.327µs) 404 [ 10.42.2.0:31631]
I0201 06:29:07.112165       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (480.506µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:07.193127       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (340.978µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:07.209336       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (327.768µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:08.207644       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (329.738µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:08.227716       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (321.087µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:08.294245       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (482.517µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:09.546488       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:10.605201       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:13.610463       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (403.278µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.610655       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (425.307µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.610695       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (574.016µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.610784       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (383.128µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.610864       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (433.687µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.612192       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (319.737µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.612836       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (262.808µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.613881       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (257.448µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.614377       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (253.958µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.614614       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (197.079µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:13.823267       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.630945ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:13.896414       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (504.877µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:29:13.917944       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (321.688µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:14.546593       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:14.638599       1 wrap.go:47] GET /openapi/v2: (3.408136ms) 404 [ 10.42.0.0:1995]
I0201 06:29:14.640526       1 wrap.go:47] GET /openapi/v2: (336.188µs) 404 [ 10.42.0.0:1995]
I0201 06:29:15.008611       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (332.738µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:15.090890       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (330.498µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:15.109880       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (328.178µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:15.605326       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:17.793543       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:17.793588       1 event_handlers.go:225] [UpdatePod] Updated pod <haste/pipeline-client-7c8cc9d588-n78dk> in cache.
I0201 06:29:18.089051       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (304.328µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.089231       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (302.649µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.089363       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (393.518µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.089447       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (265.768µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.089745       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (226.979µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.093047       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (414.767µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.093319       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (443.417µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.093335       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (744.824µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.093443       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (320.048µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.093925       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (533.446µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:18.389098       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.722303ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:29:18.438139       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.299076ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:29:19.546732       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:20.605456       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:20.720102       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (350.428µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:20.737854       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (430.887µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:20.789936       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (359.077µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:20.867480       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.180528ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:29:20.967273       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.209377ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:29:21.897695       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (395.187µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:21.921500       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (337.658µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:21.991166       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (308.428µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:22.793684       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:24.546868       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:25.605597       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:27.793853       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:28.148879       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.732683ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:28.171500       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (415.997µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:28.197656       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (476.146µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:29.223863       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (307.278µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:29.299612       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (298.957µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:29.315778       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (295.008µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:29.547018       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:30.324287       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.484465ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:29:30.373611       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.103969ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:29:30.605752       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:32.794042       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:34.547142       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:35.005513       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (465.597µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:35.027727       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (356.847µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:29:35.093640       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (254.859µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:35.605887       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:36.109242       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (310.037µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109341       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (275.698µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109568       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (456.147µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109734       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (410.867µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109734       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (451.897µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109744       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (210.548µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109736       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (294.068µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.109985       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (256.818µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.110076       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (261.178µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.110416       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (417.627µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:29:36.207339       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (453.667µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:36.290986       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (302.617µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:29:36.312879       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (470.786µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:37.111557       1 wrap.go:47] GET /openapi/v2: (2.91ms) 404 [ 10.42.2.0:31631]
I0201 06:29:37.130913       1 wrap.go:47] GET /openapi/v2: (313.567µs) 404 [ 10.42.2.0:31631]
I0201 06:29:37.794218       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:39.547274       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:40.606005       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:42.009664       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.135801ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:42.031652       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (407.597µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:42.104647       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (343.608µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:42.794384       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:43.105166       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (470.657µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:43.205891       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (346.148µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:43.224910       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (302.728µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:43.610747       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (392.347µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.610992       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (291.838µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.611343       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (322.038µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.611647       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (334.827µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.612027       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (232.579µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.612066       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (243.278µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.612292       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (626.936µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.612573       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (507.066µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.612700       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (249.498µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:43.614403       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (261.188µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:29:44.547403       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:45.606129       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:47.794574       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:48.090627       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (296.568µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.091012       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (443.397µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.091016       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (598.386µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.091461       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (277.458µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.093047       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (340.208µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.093336       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (210.299µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.093859       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (280.398µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.094789       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (292.457µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.094968       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (213.559µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.095165       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (219.219µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:29:48.706476       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (297.987µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:48.722995       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (311.938µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:48.793564       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (308.678µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:48.843191       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.947209ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:29:48.893256       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.900419ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:29:49.547496       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:49.808065       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (461.146µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:49.829747       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (287.998µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:49.892516       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (311.868µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:50.606258       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:52.399312       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.907112ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:29:52.547836       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.443436ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:29:52.794749       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:54.547624       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:55.606371       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:55.704385       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.407986ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:55.736634       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (200.069µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:55.789142       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (335.807µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:56.920297       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (317.037µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:56.949880       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (339.778µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:56.992794       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (481.677µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:29:57.794877       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:29:59.547716       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:00.606500       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:02.778568       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.834303ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:30:02.795028       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:02.827897       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.037399ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:30:02.997480       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (465.357µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:03.022447       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (460.377µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:03.037986       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (312.887µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:04.097766       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (216.669µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:04.121831       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (559.336µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:04.195373       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (385.937µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:04.547844       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:05.606642       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:06.109236       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (406.417µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109355       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (322.088µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109406       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (539.766µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109415       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (603.386µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109430       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (417.537µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109418       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (632.466µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109503       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (414.877µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.109571       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (475.607µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.110170       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (279.798µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:06.110398       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (436.657µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:07.795148       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:09.548005       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:10.001884       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.236457ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:10.023504       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (331.427µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:10.092175       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (389.647µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:10.606766       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:11.107408       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (412.877µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:30:11.128775       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (454.877µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:11.259989       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (281.808µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:12.795316       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:13.611139       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (265.589µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.611830       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (222.179µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.611998       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (270.178µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.612399       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (506.977µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.612456       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (346.587µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.614365       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (400.257µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.614531       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (370.798µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.614750       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (292.168µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.614910       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (291.408µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:13.615550       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (586.636µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:14.548147       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:15.606920       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:16.906413       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (369.107µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:16.983744       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (299.558µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:17.113764       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (487.157µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.2.0:31631]
I0201 06:30:17.117356       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.171087ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.2.0:31631]
I0201 06:30:17.795479       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:18.090388       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (351.078µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.090908       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (334.858µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.091387       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (542.176µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.091760       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (320.508µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.093857       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (290.958µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.093856       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (461.657µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.093984       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (757.325µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.094004       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (265.148µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.094253       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (205.988µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.094505       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (302.998µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:18.418256       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (325.427µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:18.440719       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (325.008µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:18.490561       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (302.857µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:19.548287       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:20.607061       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:21.449043       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.312687ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:30:21.547258       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.439066ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:30:22.795606       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:24.028594       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.266527ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:30:24.078346       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.201848ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:30:24.419074       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.81259ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:24.441297       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (304.867µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:24.496161       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (309.757µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:24.548441       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:25.598755       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (311.648µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:25.607247       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:25.618721       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (507.367µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:25.642367       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (406.627µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:27.795737       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:29.548600       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:30.607400       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:31.524000       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (470.257µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:31.591294       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (339.797µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:31.622088       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (282.838µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:32.711855       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (305.478µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:32.789484       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (480.287µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:32.795888       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:32.811875       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (295.918µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:33.133891       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.922252ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:30:33.882847       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.207818ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:30:34.548714       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:35.607542       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:36.109337       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (265.248µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.109363       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (321.177µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.109337       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (241.208µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.109476       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (259.078µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.109505       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (377.298µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.109766       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (220.589µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.110036       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (405.867µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.110564       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (201.669µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.110695       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (276.018µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:36.111483       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (483.487µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:30:37.116403       1 wrap.go:47] GET /openapi/v2: (3.297987ms) 404 [ 10.42.2.0:31631]
I0201 06:30:37.136945       1 wrap.go:47] GET /openapi/v2: (318.668µs) 404 [ 10.42.2.0:31631]
I0201 06:30:37.796031       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:38.606458       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.371653ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:38.623341       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (289.188µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:38.643332       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (296.718µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:39.548851       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:39.707650       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (309.427µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:39.790083       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (291.728µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:39.813558       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (304.198µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:40.607697       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:42.796172       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:43.611098       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (271.858µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.611379       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (386.487µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.611386       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (292.048µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.611379       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (279.978µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.611494       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (427.437µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.613271       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (373.517µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.613402       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (261.288µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.613615       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (343.648µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.613802       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (286.248µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:43.614100       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (276.028µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:30:44.548936       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:45.607788       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:45.697652       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (302.247µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:45.721866       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (316.968µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:30:45.741170       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (388.607µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:46.904993       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (374.148µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:46.929884       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (389.278µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:46.992038       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (410.257µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:30:47.796295       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:48.091588       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (377.127µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.091788       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (341.178µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.091994       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (264.538µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.092809       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (510.336µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.093017       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (437.727µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.110930       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (274.847µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.111324       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (441.967µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.111673       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (349.627µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.111844       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (215.069µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:48.112408       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (263.618µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:30:49.549119       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:50.607936       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:52.701058       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.070442ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:30:52.725637       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (448.806µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:52.745440       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (370.347µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:52.796408       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:52.953665       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.172347ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:30:53.003515       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.245917ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:30:53.806661       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (425.357µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:30:53.827831       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (309.018µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:53.846197       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (300.547µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:54.086711       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:54.086749       1 event_handlers.go:225] [UpdatePod] Updated pod <serving/junk-uey-minio-576749b6b8-zjwbd> in cache.
I0201 06:30:54.549236       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:55.553594       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.139238ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:30:55.608077       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:55.653292       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.759881ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:30:57.796567       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:59.086920       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:59.549361       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:30:59.693517       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (449.187µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:59.730302       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (431.266µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:30:59.793272       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (294.908µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:00.608203       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:00.813854       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (326.337µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:00.836150       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (427.897µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:00.897583       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (323.918µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:02.796708       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:04.087083       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:04.298102       1 wrap.go:47] GET /openapi/v2: (3.963022ms) 404 [ 10.42.3.0:55048]
I0201 06:31:04.314571       1 wrap.go:47] GET /openapi/v2: (300.658µs) 404 [ 10.42.3.0:55048]
I0201 06:31:04.549465       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:05.336461       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.189978ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:31:05.536090       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.953698ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:31:05.608324       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:05.878697       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:05.878715       1 event_handlers.go:225] [UpdatePod] Updated pod <serving/junk-uey-minio-576749b6b8-zjwbd> in cache.
I0201 06:31:06.109702       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (193.658µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.109850       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (314.327µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.109981       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (299.078µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110032       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (305.048µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110216       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (467.677µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110230       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (342.088µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110440       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (403.527µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110469       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (227.938µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110559       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (286.247µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.110783       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (273.468µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:06.699896       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.305387ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:06.723673       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (205.238µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:06.743939       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (330.617µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:07.796839       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:07.931403       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (173.379µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:07.951759       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (298.597µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:07.994645       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (381.307µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:09.087207       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:09.549663       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:10.608470       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:10.878776       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:12.796952       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:13.611821       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (431.287µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.611878       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (553.506µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.611828       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (350.488µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.611828       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (468.807µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.612261       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (429.908µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.612311       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (219.538µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.612462       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (418.927µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.612462       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (376.857µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:13.612579       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (141.869µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.613433       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (462.656µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.613449       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (285.078µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:13.630387       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (244.988µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:13.694300       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (263.388µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:14.087318       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:14.549809       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:14.613706       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (308.747µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:14.659710       1 wrap.go:47] GET /openapi/v2: (3.528406ms) 404 [ 10.42.0.0:1995]
I0201 06:31:14.661241       1 wrap.go:47] GET /openapi/v2: (306.598µs) 404 [ 10.42.0.0:1995]
I0201 06:31:14.696060       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (314.198µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:14.717586       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (339.978µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:15.608611       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:15.878901       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:17.797085       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:18.092775       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (422.387µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.092784       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (392.937µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.093241       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (545.477µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.093362       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (347.028µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.093430       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (558.526µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.094240       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (263.318µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.094452       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (431.317µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.094568       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (207.068µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.094827       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (218.079µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:18.095062       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (199.879µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:19.087457       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:19.093064       1 wrap.go:47] GET /openapi/v2: (333.558µs) 404 [ 10.42.3.0:55048]
I0201 06:31:19.098305       1 wrap.go:47] GET /openapi/v2: (277.727µs) 404 [ 10.42.3.0:55048]
I0201 06:31:19.549911       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:20.406076       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.756531ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:20.428845       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (374.227µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:20.491151       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (262.618µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:20.608700       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:20.879054       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:21.412606       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (290.748µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:21.527550       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (340.038µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:21.599695       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (316.767µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:22.797248       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:24.087605       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:24.315510       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.552265ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:31:24.550033       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:24.665799       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.953382ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:31:25.608816       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:25.879206       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:27.215123       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.971159ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:31:27.265106       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.938579ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:31:27.394711       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (297.538µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:27.425930       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (383.987µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:27.512360       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (271.998µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:27.797390       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:28.492042       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (331.208µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:28.514091       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (279.298µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:28.534863       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (455.036µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:29.087740       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:29.550161       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:30.608974       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:30.879365       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:32.797561       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:34.087863       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:34.116530       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.334906ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:34.191277       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (492.086µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:34.211741       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (329.678µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:34.550292       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:35.304807       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (270.048µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:35.329827       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (300.808µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:35.398542       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (340.537µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:35.609172       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:35.879505       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:36.110253       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (336.178µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.110360       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (418.627µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.110793       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (423.967µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.110854       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (277.398µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.110939       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (436.227µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.111148       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (458.976µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.111231       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (432.897µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.111348       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (222.608µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.111376       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (200.249µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:36.111400       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (400.468µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:31:37.491986       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (4.281009ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:31:37.541016       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.323006ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:31:37.797732       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:39.088028       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:39.550420       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:40.609308       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:40.879629       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:42.797871       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:43.612527       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (428.357µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.612539       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (430.677µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.612602       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (530.616µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.612646       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (354.607µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.612677       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (380.247µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.612983       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (385.067µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.612983       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (402.457µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.614130       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (299.348µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.614380       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (329.967µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.615196       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (311.289µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:31:43.916418       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.115478ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:31:43.960874       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.744801ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:31:44.088187       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:44.550546       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:45.609391       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:45.879779       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:47.798021       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:48.093471       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (536.076µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.093523       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (564.985µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.093634       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (172.049µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.093648       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (338.478µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.093705       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (515.706µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.094028       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (278.268µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.094201       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (369.848µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.094736       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (243.929µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.095358       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (262.368µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.096097       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (144.069µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:31:48.995503       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.120698ms) 200 [kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50 10.42.2.0:31631]
I0201 06:31:48.996395       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.076186ms) 200 [kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50 10.42.3.0:55048]
I0201 06:31:49.088313       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:49.550656       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:50.462879       1 queuejob_controller_ex.go:943] [Informer-addQJ] enqueue stateful-set-2-replicas &qj=0xc003769880 Version=335269358 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State: Message: SystemPriority:0 QueueJobState:Init ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:false Sender: Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:}]}
I0201 06:31:50.463044       1 queuejob_controller_ex.go:1189] [Agent Mode] Deleting resources for XQJ stateful-set-2-replicas because it will be preempted (newjob)
I0201 06:31:50.463075       1 queuejob_controller_ex.go:1362] [Cleanup] begin AppWrapper stateful-set-2-replicas Version=335269358 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State: Message: SystemPriority:0 QueueJobState:Init ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:false Sender: Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:}]}
I0201 06:31:50.474440       1 queuejob_controller_ex.go:1385] [Cleanup] end   AppWrapper stateful-set-2-replicas Version=335269358 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State: Message: SystemPriority:0 QueueJobState:Init ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:false Sender: Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:}]}
I0201 06:31:50.494527       1 queuejob_controller_ex.go:1212] [worker-manageQJ] stateful-set-2-replicas 1Delay=0.031650 seconds activeQ.Add_success activeQ=true Unsched=false &qj=0xc003769880 Version=335269359 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State:Pending Message: SystemPriority:0 QueueJobState:Queueing ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:true Sender:after  [manageQueueJob]setQueueing Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465505 +0000 UTC m=+229.188480826 LastTransitionMicroTime:2021-02-01 06:31:50.474465574 +0000 UTC m=+229.188480886 Reason:AwaitingHeadOfLine Message:}]}
I0201 06:31:50.494607       1 queuejob_controller_ex.go:623] [ScheduleNext] activeQ.Pop stateful-set-2-replicas *Delay=0.031732 seconds RemainingLength=0 &qj=0xc003769880 Version=335269359 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State:Pending Message: SystemPriority:0 QueueJobState:Queueing ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:true Sender:after  [manageQueueJob]setQueueing Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465505 +0000 UTC m=+229.188480826 LastTransitionMicroTime:2021-02-01 06:31:50.474465574 +0000 UTC m=+229.188480886 Reason:AwaitingHeadOfLine Message:}]}
I0201 06:31:50.520632       1 queuejob_controller_ex.go:687] [ScheduleNext] [Agent Mode] Deploy Next QueueJob: stateful-set-2-replicas Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State:Pending Message: SystemPriority:0 QueueJobState:HeadOfLine ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:true Sender:after  ScheduleNext - setHOL Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465505 +0000 UTC m=+229.188480826 LastTransitionMicroTime:2021-02-01 06:31:50.474465574 +0000 UTC m=+229.188480886 Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659481 +0000 UTC m=+229.208674843 LastTransitionMicroTime:2021-02-01 06:31:50.494659661 +0000 UTC m=+229.208675013 Reason:FrontOfQueue. Message:}]}
I0201 06:31:50.520862       1 queuejob_controller_ex.go:508] [getAggAvaiResPri] Idle cluster resources cpu 201600.00, memory 1240504110592.00, GPU 1
I0201 06:31:50.520892       1 queuejob_controller_ex.go:588] [getAggAvaiResPri] cpu 201600.00, memory 1240504110592.00, GPU 1 available resources to schedule
I0201 06:31:50.520901       1 queuejob_controller_ex.go:735] [ScheduleNext] XQJ stateful-set-2-replicas with resources cpu 0.00, memory 419430400.00, GPU 0 to be scheduled on aggregated idle resources cpu 201600.00, memory 1240504110592.00, GPU 1
I0201 06:31:50.581971       1 queuejob_controller_ex.go:767] [ScheduleNext] stateful-set-2-replicas Delay=0.119100 seconds eventQueue.Add_afterHeadOfLine activeQ=false, Unsched=false &qj=0xc003769880 Version=335269361 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Pending Message: SystemPriority:0 QueueJobState:HeadOfLine ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:true Sender:after  ScheduleNext - setCanRun Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465505 +0000 UTC m=+229.188480826 LastTransitionMicroTime:2021-02-01 06:31:50.474465574 +0000 UTC m=+229.188480886 Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659481 +0000 UTC m=+229.208674843 LastTransitionMicroTime:2021-02-01 06:31:50.494659661 +0000 UTC m=+229.208675013 Reason:FrontOfQueue. Message:}]}
I0201 06:31:50.582023       1 queuejob_controller_ex.go:1234] [worker-manageQJ] stateful-set-2-replicas 3Delay=0.119155 seconds BeforeDispatchingToEtcd Version=335269361 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:HeadOfLine ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:true Sender:after  ScheduleNext - setCanRun Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465505 +0000 UTC m=+229.188480826 LastTransitionMicroTime:2021-02-01 06:31:50.474465574 +0000 UTC m=+229.188480886 Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659481 +0000 UTC m=+229.208674843 LastTransitionMicroTime:2021-02-01 06:31:50.494659661 +0000 UTC m=+229.208675013 Reason:FrontOfQueue. Message:}]}
I0201 06:31:50.587445       1 statefulset.go:237] QJob: stateful-set-2-replicas had 0 StatefulSets and 1 desired StatefulSets
I0201 06:31:50.587491       1 statefulset.go:193] ==========create statefulSet: default,  &StatefulSet{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:stateful-set-2-replicas,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: stateful-set-2-replicas,appwrapper.mcad.ibm.com: stateful-set-2-replicas,qjobResLabel: 0,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:StatefulSetSpec{Replicas:*2,Selector:&k8s_io_apimachinery_pkg_apis_meta_v1.LabelSelector{MatchLabels:map[string]string{app: stateful-set-2-replicas,},MatchExpressions:[],},Template:k8s_io_api_core_v1.PodTemplateSpec{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: stateful-set-2-replicas,appwrapper.mcad.ibm.com: stateful-set-2-replicas,size: 2,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[],Containers:[{stateful-set-2-replicas k8s.gcr.io/echoserver:1.4 [] []  [] [] [] {map[memory:{{209715200 0} {<nil>}  BinarySI}] map[memory:{{209715200 0} {<nil>}  BinarySI}]} [] [] nil nil nil    nil false false false}],RestartPolicy:,TerminationGracePeriodSeconds:nil,ActiveDeadlineSeconds:nil,DNSPolicy:,NodeSelector:map[string]string{},ServiceAccountName:,DeprecatedServiceAccount:,NodeName:,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:nil,ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[],HostAliases:[],PriorityClassName:,Priority:nil,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,},},VolumeClaimTemplates:[],ServiceName:,PodManagementPolicy:,UpdateStrategy:StatefulSetUpdateStrategy{Type:,RollingUpdate:nil,},RevisionHistoryLimit:nil,},Status:StatefulSetStatus{ObservedGeneration:0,Replicas:0,ReadyReplicas:0,CurrentReplicas:0,UpdatedReplicas:0,CurrentRevision:,UpdateRevision:,CollisionCount:nil,Conditions:[],},}
I0201 06:31:50.609529       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:50.630167       1 statefulset.go:224] Finished syncing queue job resource stateful-set-2-replicas (48.064211ms)
I0201 06:31:50.630187       1 queuejob_controller_ex.go:1266] [worker-manageQJ] stateful-set-2-replicas 4Delay=0.167316 seconds AllResourceDispatchedToEtcd Version=335269361 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865836 +0000 UTC m=+229.176881228 FilterIgnore:true Sender:after  ScheduleNext - setCanRun Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866656 +0000 UTC m=+229.176882008 LastTransitionMicroTime:2021-02-01 06:31:50.462866796 +0000 UTC m=+229.176882148 Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465505 +0000 UTC m=+229.188480826 LastTransitionMicroTime:2021-02-01 06:31:50.474465574 +0000 UTC m=+229.188480886 Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659481 +0000 UTC m=+229.208674843 LastTransitionMicroTime:2021-02-01 06:31:50.494659661 +0000 UTC m=+229.208675013 Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181425 +0000 UTC m=+229.344196777 LastTransitionMicroTime:2021-02-01 06:31:50.630181575 +0000 UTC m=+229.344196927 Reason:AppWrapperRunnable Message:}]}
I0201 06:31:50.879910       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:51.846534       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=1.383657 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:31:51.846789       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:31:52.798140       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:54.088435       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:54.311486       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.272057ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:54.332937       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (130.299µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:54.397333       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (423.477µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:54.550783       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:55.521280       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (174.208µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:31:55.545701       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (291.068µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:55.594756       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (283.838µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:31:55.609659       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:55.880020       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:56.219012       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.406864ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:31:56.269015       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.455603ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:31:56.846766       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=6.383885 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:31:56.846960       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:31:57.798268       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:59.088602       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:31:59.550922       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:00.609783       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:00.880143       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:01.195750       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (430.497µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:01.226086       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (440.207µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:01.271142       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (270.998µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:01.847080       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=11.384191 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:01.847233       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:02.392583       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (303.408µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:02.437613       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (262.368µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:02.454925       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (167.369µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:02.798383       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:04.088703       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:04.551018       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:05.609871       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:05.880250       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:06.110579       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (490.767µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110613       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (365.878µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110628       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (362.997µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110680       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (344.178µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110721       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (310.678µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110842       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (327.898µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110911       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (453.546µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.110958       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (565.916µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.111000       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (491.786µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.111061       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (513.646µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:06.847523       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:06.847426       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=16.384467 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:07.798516       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:08.102196       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.882903ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:08.123793       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (351.277µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:08.143540       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (387.327µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:08.950301       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.032499ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:32:09.050521       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.232217ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:32:09.088855       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:09.311828       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (196.508µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:09.335272       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (326.688µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:09.354210       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (320.067µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:09.551144       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:10.609976       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:10.880366       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:11.847711       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=21.384830 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:11.847993       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:12.798622       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:13.612843       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (412.787µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.613109       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (415.467µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.613478       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (278.398µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.613898       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (217.389µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.614301       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (247.238µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.614708       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (220.749µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.614845       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (587.056µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.614865       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (294.218µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.615122       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (210.279µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:13.615806       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (258.058µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:14.088993       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:14.551267       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:15.390651       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.267416ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:32:15.491184       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.273947ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/controller-discovery 10.42.3.0:55048]
I0201 06:32:15.521077       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (349.658µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:15.543825       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (350.967µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:15.593603       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (341.348µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:15.610133       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:15.880469       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:16.694632       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (497.486µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:16.716248       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (302.678µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:16.733928       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (320.338µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:16.848176       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:16.848006       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=26.385123 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:17.798777       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:18.094491       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (505.317µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.094675       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (679.235µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.095623       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (233.648µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.095729       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (149.679µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.095867       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (107.69µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.096049       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (101.38µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.096297       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (150.099µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.096449       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (112.929µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.097960       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (163.259µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:18.098088       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (149.249µs) 200 [Go-http-client/2.0 10.42.3.0:47539]
I0201 06:32:19.089132       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:19.099047       1 wrap.go:47] GET /openapi/v2: (3.829403ms) 404 [ 10.42.3.0:55048]
I0201 06:32:19.108304       1 wrap.go:47] GET /openapi/v2: (300.988µs) 404 [ 10.42.3.0:55048]
I0201 06:32:19.551377       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:20.610300       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:20.880595       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:21.848431       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=31.385534 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:21.848627       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:22.505251       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.502995ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:22.526951       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (468.107µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:22.589863       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (314.208µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:22.798930       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:23.650173       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (259.099µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:23.695564       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (324.508µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:23.790302       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (311.438µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:24.089256       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:24.551478       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:25.610532       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:25.880719       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:26.848876       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=36.385990 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:26.849095       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:27.579931       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (3.980042ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:32:27.799118       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:27.829573       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.190838ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:generic-garbage-collector 10.42.3.0:55048]
I0201 06:32:29.089484       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:29.522134       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (339.337µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:29.545990       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (270.929µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:29.551617       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:29.563893       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (359.368µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:30.610688       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:30.702024       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (455.366µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:30.728560       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (415.627µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:30.749522       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (351.487µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:30.880848       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:31.849216       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=41.386328 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:31.849459       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:32.799347       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:34.089634       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:34.551801       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:35.610829       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:35.880950       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:36.111187       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (402.758µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111334       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (351.687µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111440       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (414.587µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111509       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (409.267µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111510       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (420.797µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111814       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (321.208µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111898       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (307.428µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111947       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (286.238µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.111993       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (230.039µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.112118       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (307.848µs) 200 [Go-http-client/2.0 10.42.2.0:62773]
I0201 06:32:36.411158       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.482175ms) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:36.429952       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (334.577µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:36.450568       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (311.898µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:36.849515       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=46.386638 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:36.849797       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:37.120167       1 wrap.go:47] GET /openapi/v2: (2.507742ms) 404 [ 10.42.2.0:31631]
I0201 06:32:37.151691       1 wrap.go:47] GET /openapi/v2: (207.498µs) 404 [ 10.42.2.0:31631]
I0201 06:32:37.506163       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (277.088µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:37.528308       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (364.507µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:37.597501       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (262.818µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:37.799496       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:39.089778       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:39.551977       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:39.804316       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (3.276657ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:32:39.954319       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1?timeout=32s: (2.81588ms) 200 [kube-controller-manager/v1.18.8 (linux/amd64) kubernetes/9f2892a/system:serviceaccount:kube-system:resourcequota-controller 10.42.3.0:55048]
I0201 06:32:40.610952       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:40.881078       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:41.849741       1 queuejob_controller_ex.go:910] [UpdateQueueJobs] stateful-set-2-replicas *Delay=51.386865 seconds eventQueue.Add_byUpdateQueueJobs &qj=0xc002fe8e00 Version=335269363 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:true IsDispatched:false State:Running Message: SystemPriority:0 QueueJobState:Dispatched ControllerFirstTimestamp:2021-02-01 06:31:50.462865 +0000 UTC FilterIgnore:true Sender:before [manageQueueJob]afterEtcdDispatching Local:true Conditions:[{Type:Init Status:True LastUpdateMicroTime:2021-02-01 06:31:50.462866 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.462866 +0000 UTC Reason: Message:} {Type:Queueing Status:True LastUpdateMicroTime:2021-02-01 06:31:50.474465 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.474465 +0000 UTC Reason:AwaitingHeadOfLine Message:} {Type:HeadOfLine Status:True LastUpdateMicroTime:2021-02-01 06:31:50.494659 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.494659 +0000 UTC Reason:FrontOfQueue. Message:} {Type:Dispatched Status:True LastUpdateMicroTime:2021-02-01 06:31:50.630181 +0000 UTC LastTransitionMicroTime:2021-02-01 06:31:50.630181 +0000 UTC Reason:AppWrapperRunnable Message:}]}
I0201 06:32:41.849922       1 pod.go:236] [UpdateQueueJobStatus] There are 0 pods of QueueJob stateful-set-2-replicas:  pending 0, running 0, succeeded 0, failed 0
I0201 06:32:42.799626       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:43.397991       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (410.747µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.3.0:55048]
I0201 06:32:43.471466       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (299.278µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:43.491692       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (326.868µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:43.613527       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (364.588µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.614380       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (303.258µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.614709       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (250.059µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.614997       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (350.727µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.615024       1 wrap.go:47] GET /apis/external.metrics.k8s.io/v1beta1: (240.328µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.615601       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (215.718µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.615905       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (301.218µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.615911       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (259.618µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.616103       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (356.158µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:43.616341       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1: (409.497µs) 200 [Go-http-client/2.0 10.42.0.0:57984]
I0201 06:32:44.089895       1 cache.go:240] [deleteJob] Attempting to delete Job <4efead19-557b-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:44.507838       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (320.028µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:44.552121       1 cache.go:240] [deleteJob] Attempting to delete Job <8efd2f4f-5ece-11eb-9caf-ac1f6b8d1716:/>
I0201 06:32:44.597203       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (349.347µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:44.619243       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (367.198µs) 200 [kubectl/v1.20.1 (linux/amd64) kubernetes/c4d7527 10.42.0.0:1995]
I0201 06:32:45.016138       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.634971ms) 200 [kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50 10.42.0.0:1995]
I0201 06:32:45.059422       1 wrap.go:47] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (370.807µs) 200 [kubectl/v1.16.2 (linux/amd64) kubernetes/c97fe50 10.42.0.0:1995]

Add timeout to genericItems in Appwrapper

This feature could be modeled after Kubernetes batchv1 job feature called activedeadlineseconds. The idea is to check the duration when appwrapper changes to Running state. Once duration == userTimeout, we kill all the items in the appwrapper and generated a new state called FailedDeadlineExceeded.

[testing] Add test case to put jobs back into the queue after MCAD restart

When MCAD is restarted (or redeployed) a test case is needed to make sure all the queued jobs are added to the MCAD queue

Add https clone link to build instructions

Current docs show how to clone using ssh url. Need to add https option as well. See: https://github.com/IBM/multi-cluster-app-dispatcher/blob/master/doc/build/build.md#clone-multi-cluster-app-deployer-git-repo

Add a link to deployment docs at the end of the build docs.

For usability purposes, add a link to the deployment document at the end of the build document page: https://github.com/IBM/multi-cluster-app-dispatcher/blob/master/doc/build/build.md#build-the-multi-cluster-app-dispatcher. For reference the deployment document is here: https://github.com/IBM/multi-cluster-app-dispatcher/blob/master/doc/deploy/deployment.md

[core] AppWrapper CRD should specify a status printer column

It would be nice if kubectl get appwrapper had a status column.

On Dispatcher mode agent fails to load with undefined cordon definition.

agentConfigs='agent110config' without a uncordon/cordon setting fails to load in dispatcher.

Example use with spark-submit

does anyone have an example or doc to use with spark-submit?

Readme reference incorrect repo name.

readme.md reference multi-cluster-app-wrapper incorrectly. Should be multi-cluster-app-dispatcher.

How to specify a kubernetes Job resource in AppWrapper

I would like to specify a kubernetes Job (apiVersion: batch/v1) as an AppWrapper but don't find documentation on how to do it.
Is it possible?

Update Operator CRD Definitons to v1

For example ./deployment/mcad-operator/helm-charts/mcad-controller/templates/deployment.yaml
Change

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: appwrappers.mcad.ibm.com
spec:
  group: mcad.ibm.com
  names:
    kind: AppWrapper
    listKind: AppWrapperList
    plural: appwrappers
    singular: appwrapper
  scope: Namespaced
  version: v1alpha1

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: appwrappers.mcad.ibm.com
spec:
  group: mcad.ibm.com
  names:
    kind: AppWrapper
    listKind: AppWrapperList
    plural: appwrappers
    singular: appwrapper
  scope: Namespaced
  versions: 
    - name: v1alpha1
      served: true
      storage: true

And upgrade operator.

Removed unused Job Spec

Reference: /github.com/IBM/multi-cluster-app-dispatcher/pkg/apis/controller/v1alpha1/queuejob.go

Propagate running job status into MCAD

MCAD needs the ability to propagate job running state. completion status PR: #201 could handle such requirement.

Remove depricated client objects from opensource repo.

Deprecated client objects to be removed from: pkg/client/clientset/controller-versioned/clients

Support for pod

Current code does not support the following request (appwrapper wrapped pod) properly:
apiVersion: arbitrator.incubator.k8s.io/v1alpha1
kind: AppWrapper
metadata:
name: nginx
spec:
schedSpec:
minAvailable: 2
priority: 3
resources:
Items:
- replicas: 1
type: Pod
template:
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
resources:
limits:
cpu: 125m
memory: 1024Mi
requests:
cpu: 125m
memory: 1024Mi

pod object needs namespace in metadata section

The issue with Kind Pod as reported here: #190 needs explicit mention of namespace in metadata section. we need to inform users about errors when selector fails in pod.go file

Support build with golang v1.16 image.

Error building with golang v1.16 image:
go: cannot find main module, but found .git/config in /go/src/github.com/IBM/multi-cluster-app-dispatcher
to create a module there, run:
go mod init
make: *** [Makefile:53: generate-code] Error 1

Addition of new states to Appwrapper

Additional Appwrapper states are needed to take clean-up actions. Clean-up actions can be based on complete or Failed state of any object running inside GenericItem block. Users would need a way to express such conditions to trigger additional actions like clean-up.

[Doc] Updated documentation on quota-management branch for helm3 install

Reference PR: #206

Rename folders to reflect opensource rename changes

For example: github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/queuejobresources should be github.com/IBM/multi-cluster-app-dispatcher/pkg/controller/appwrapperresources

Helm install fails

Hi,
I was trying to install MCAD on an OpenShift cluster with three 4cpux16Gb worker nodes.
I built the image and pushed it into my quay.io repository.
This is the output I get from Helm

root@kubeflux-jump-1:~/git/multi-cluster-app-dispatcher/deployment# helm install mcad-controller --namespace kube-system --generate-name --set image.repository=quay.io/cmisale1/mcad-controller --set image.tag=master-v1.29.37
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/KUBECONF
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/KUBECONF
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "schedulingspecs.mcad.ibm.com" namespace: "" from "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first, resource mapping not found for name: "queuejobs.mcad.ibm.com" namespace: "" from "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first, resource mapping not found for name: "xqueuejobs.mcad.ibm.com" namespace: "" from "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first, resource mapping not found for name: "appwrappers.mcad.ibm.com" namespace: "" from "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first]

Can anyone help?

Thanks

Add verification of deletion in E2E testing of Namespace only use case.

Simplify conditions check in queue_job_controller_ex.go

	```if qj.Status.CanRun && qj.Status.State != arbv1.AppWrapperStateActive &&
		qj.Status.State != arbv1.AppWrapperStateCompleted &&
		qj.Status.State != arbv1.AppWrapperStateRunningHoldCompletion {``

the idea is to remove:

qj.Status.State != arbv1.AppWrapperStateCompleted &&
    qj.Status.State != arbv1.AppWrapperStateRunningHoldCompletion

Update Helm chart to deploy a predefined name instead of a random deployed name.

During helm install mcad is setup to install with a random generated name. Change the Helm chart to install with a predefined name: multi-cluster-app-dispatcher.

Remove incubator ID from api group definition.

Transition AppWrapper group API to a dispatch identifier. E.g. transition to dispatcher.k8s.io. See CRD definitions in Helm Chart: multi-cluster-app-dispatcher/deployment/kube-arbitrator/templates/deployment.yaml

AppWrapper state not updated when job is requeued.

When an appwrapper job gets re-queued due to min set in scheduling spec the state is not updated to reflect the requeue.

"Insufficient resources to dispatch AppWrapper" message should clarify which resource(s) are insufficient

It would be helpful to know what MCAD is blocked on.

Need to merge original readme into new repo.

Question: Does MCAD respect taints and tolerations?

Question
Does MCAD respect taints and tolerations when dispatching jobs?
@asm582

**Observed Behavior **
A job will be dispatched even though it does not tolerate taints on nodes required to meet its resource request.

When using the co-scheduler, respective pods will remain pending, and the job can be starved by smaller jobs or jobs tolerating the taints.

**Desired Behavior **
Hold the job in the queue (wait to dispatch) until resources are available (excluding resources from nodes with taints that the job does not tolerate).

**Notes **

The term "job" above is being used generically; for this use-case, it is defined as an AppWrapper containing 1 PodGroup and 1 or more k8s batch Jobs.
gang scheduling is required for these jobs. Using the default scheduler in this scenario dooms the job to failure (a subset of pods can run, others will remain pending). Assume the co-scheduler is being used.

Arch diagram needs naming change

Architecture diagram referenced in readme needs to be updated to reflect naming change (XQueueJobController to MultiClusterAppDispatcher).

File: doc/images/xqueuejob-controller.png needs to be updated

nvidia-gpu-operator namespace issue

I am using OpenShift with Client Version: v4.2.0-alpha.0-1420-gf1f09a3, Server Version: 4.9.18, Kubernetes Version: v1.22.3+e790d7f to deploy a simple hello world job using MCAD AppWrapper.
I'm using the following hello world example file to create an appwrapper job:

apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
  name: aw-helloworld
  namespace: default
spec:
  resources:
    GenericItems:
      - replicas: 1
        generictemplate:
          apiVersion: v1
          kind: Pod
          metadata:
            name: helloworld
            labels:
              app: helloworld
          spec:
            containers:
              - name: helloworld
                image: vishakharamani/torchximgrepov1:myfirstimagepush

When you set the namespace to default, I can see the appwrapper and the pod associated getting created. However, if one sets the namespace to be nvidia-gpu-operator, only app wrapper is created and I see no pods associated with it.
to debug this I ran oc get appwrapper aw-helloworld -oyaml, we get the following output:

apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
  creationTimestamp: "2022-07-14T13:41:47Z"
  generation: 7
  name: aw-helloworld
  namespace: nvidia-gpu-operator
  resourceVersion: "29656480"
  uid: bad4de28-cd2e-425b-89b4-81dfb8594319
spec:
  resources:
    GenericItems:
    - allocated: 0
      generictemplate:
        apiVersion: v1
        kind: Pod
        metadata:
          labels:
            app: helloworld
          name: helloworld
        spec:
          containers:
          - image: vishakharamani/torchximgrepov1:myfirstimagepush
            name: helloworld
      metadata: {}
      priority: 0
      priorityslope: 0
      replicas: 1
    metadata: {}
  schedulingSpec: {}
  service:
    spec: {}
status:
  canrun: true
  conditions:
  - lastTransitionMicroTime: "2022-07-14T13:41:47.611624Z"
    lastUpdateMicroTime: "2022-07-14T13:41:47.611622Z"
    status: "True"
    type: Init
  - lastTransitionMicroTime: "2022-07-14T13:41:47.611819Z"
    lastUpdateMicroTime: "2022-07-14T13:41:47.611819Z"
    reason: AwaitingHeadOfLine
    status: "True"
    type: Queueing
  - lastTransitionMicroTime: "2022-07-14T13:41:47.629030Z"
    lastUpdateMicroTime: "2022-07-14T13:41:47.629029Z"
    reason: FrontOfQueue.
    status: "True"
    type: HeadOfLine
  - lastTransitionMicroTime: "2022-07-14T13:41:48.737855Z"
    lastUpdateMicroTime: "2022-07-14T13:41:48.737854Z"
    reason: AppWrapperRunnable
    status: "True"
    type: Dispatched
  controllerfirsttimestamp: "2022-07-14T13:41:47.611496Z"
  filterignore: true
  queuejobstate: Dispatched
  sender: before manageQueueJob - podPhaseChanges
  state: Running

Host MCAD images on github container registry

Move official MCAD image location to GitHub container registry with CI/CD functionality

Incorrect status reported intermittently

This may be some sort of race condition where the pending pods are not updated by MCAD leading to incorrect status, this needs to be investigated further, below is the observed status:

  Controllerfirsttimestamp:      2022-10-17T22:38:40.542570Z
  Filterignore:                  true
  Pending:                       1
  Queuejobstate:                 RunningHoldCompletion
  Sender:                        before [syncQueueJob] setRunningHoldCompletion
  State:                         RunningHoldCompletion
Events:                          <none>
(helm3) Abhisheks-MacBook-Pro:mcad-pod-status-3 abhishekmalvankar$ oc get pods
NAME                         READY   STATUS    RESTARTS   AGE
test-dep-job-without-4qqsx   1/1     Running   0          50s

Metric Logging Error

custom.metrics error from go log:

I0127 13:52:46.271766 1 round_trippers.go:438] GET https://MY_IP_ADDRESS:443/apis/custom.metrics.k8s.io/v1beta1?timeout=32s 404 Not Found in 1 milliseconds
I0127 13:52:46.271787 1 round_trippers.go:444] Response Headers:
I0127 13:52:46.271794 1 round_trippers.go:447] Content-Type: text/plain; charset=utf-8
I0127 13:52:46.271801 1 round_trippers.go:447] X-Content-Type-Options: nosniff
I0127 13:52:46.271807 1 round_trippers.go:447] Content-Length: 19
I0127 13:52:46.271814 1 round_trippers.go:447] Date: Mon, 27 Jan 2020 13:52:46 GMT
I0127 13:52:46.271835 1 request.go:942] Response Body: 404 page not found
I0127 13:52:46.271941 1 request.go:1144] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }

Preserve job logs

To debug it is important to preserve job logs. We need to explore the ability to preserve logs when jobs fails or completes

Operator projects using the removed APIs in k8s 1.22 requires changes.

Problem Description

Kubernetes has been deprecating API(s), which will be removed and are no longer available in 1.22. Operators projects using these APIs versions will not work on Kubernetes 1.22 or any cluster vendor using this Kubernetes version(1.22), such as OpenShift 4.9+. Following the APIs that are most likely your projects to be affected by:

apiextensions.k8s.io/v1beta1: (Used for CRDs and available since v1.16)
rbac.authorization.k8s.io/v1beta1: (Used for RBAC/rules and available since v1.8)
admissionregistration.k8s.io/v1beta1 (Used for Webhooks and available since v1.16)

Therefore, looks like this project distributes solutions in the repository and does not contain any version compatible with k8s 1.22/OCP 4.9. (More info). Following some findings by checking the distributions published:

mcad-operator.v0.1.9: this distribution is using APIs which were deprecated and removed in v1.22. More info: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22. Migrate the API(s) for CRD: (["mcadhelmconfigs.mcad.ibm.com"])

NOTE: The above findings are only about the manifests shipped inside of the distribution. It is not checking the codebase.

How to solve

It would be very nice to see new distributions of this project that are no longer using these APIs and so they can work on Kubernetes 1.22 and newer and published in the community-operators collection. OpenShift 4.9, for example, will not ship operators anymore that do still use v1beta1 extension APIs.

Due to the number of options available to build Operators, it is hard to provide direct guidance on updating your operator to support Kubernetes 1.22. Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the latest versions of these APIs (all that is generated by tools only). See the guides to upgrade your projects with OperatorSDK Golang, Ansible, Helm or the Kubebuilder one. For APIs other than the ones mentioned above, you will have to check your code for usage of removed API versions and upgrade to newer APIs. The details of this depend on your codebase.

If this projects only need to migrate the API for CRDs and it was built with OperatorSDK versions lower than 1.0.0 then, you maybe able to solve it with an OperatorSDK version >= v0.18.x < 1.0.0:

$ operator-sdk generate crds --crd-version=v1
INFO[0000] Running CRD generator.
INFO[0000] CRD generation complete.

Alternatively, you can try to upgrade your manifests with controller-gen (version >= v0.4.1) :

If this project does not use Webhooks:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..."

If this project is using Webhooks:

Add the markers sideEffects and admissionReviewVersions to your webhook (Example with sideEffects=None and admissionReviewVersions={v1,v1beta1}: memcached-operator/api/v1alpha1/memcached_webhook.go):
Run the command:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..."

For further information and tips see the comment.

Rename readme image file to new name for public repo.

Rename image: https://github.com/IBM/multi-cluster-app-dispatcher/blob/master/doc/images/xqueuejob-controller.png to reflect new name of public repo. Suggested name multi-cluster-app-dispatcher-controller.png. Also rename the link name, suggestion: [multi-cluster-app-dispatcher-controller]

Rename Helm deployment folder to opensource name and update deployment docs.

rename old Helm folder name in deployment to new Helm name for opensource. New folder name: mcad.

question: pods in a completed state count against available-resource-calculation?

I understand that pods in a completed (Succeeded) state count against MCAD's available-resources calculation. Is that correct? If so, what is the rationale, or is this aspect of the design changeable (open to a future feature request)?

One motivation is fully supporting Jobs, for which there can be value to keep completed pods around for a while (albeit only a short while) https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/, without unnecessarily blocking other enqueued workloads.

Folder housing old name for binary should be updated to new name.

kar-controller needs to updated to new name: mcad-controller, reference: https://github.com/IBM/multi-cluster-app-dispatcher/tree/master/cmd/kar-controllers

Update deployment documentation to reflect open source project name and github.

Reference: https://github.com/IBM/multi-cluster-app-dispatcher/blob/master/doc/deploy/deployment.md

Add instructions to push built image to a docker registry in the build documentation.

Additional instructions should be put in this document after the identified section: https://github.com/IBM/multi-cluster-app-dispatcher/blob/master/doc/build/build.md#build-the-multi-cluster-app-dispatcher

Avoid adding nodes that are not ready to schedule jobs

To implement this feature:

Update NodeInfo struct with IsReady v1.ConditionStatus Reference: https://kubernetes.io/docs/concepts/architecture/nodes/#condition
Filter cluster nodes by following condition:

for _, value := range cluster.Nodes {
		// Do not use any Unschedulable nodes in calculations
		if !value.Unschedulable {
			if value.IsReady == "true" {
				klog.V(6).Infof("[updateState] %s is marked as unschedulable or not ready node Total: %v, Used: %v, and Idle: %v will not be included in cluster state calculation.",
					value.Name, value.Allocatable, value.Used, value.Idle)
				continue
			}
		}

Missing Windows dependency

github.com/Azure/go-ansiterm
and
github.com/sirupsen/logrus

AppWrapper CRD Forbidden Error

I0127 13:52:46.274821 1 builder.go:111] Entered initFlagSet()
I0127 13:52:46.274832 1 builder.go:122] initFlagSet(): Error getting secure-port:
I0127 13:52:46.274840 1 builder.go:124] initFlagSet(): Flagset secure-port = 6443
I0127 13:52:46.291232 1 queuejob_controller_ex.go:348] [Controller] Agent mode
I0127 13:52:46.291373 1 request.go:942] Request Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1beta1","metadata":{"name":"appwrappers.arbitrator.incubator.k8s.io","creationTimestamp":null},"spec":{"group":"arbitrator.incubator.k8s.io","version":"v1alpha1","names":{"plural":"appwrappers","kind":"AppWrapper"},"scope":"Namespaced"},"status":{"conditions":null,"acceptedNames":{"plural":"","kind":""},"storedVersions":null}}
I0127 13:52:46.291469 1 round_trippers.go:419] curl -k -v -XPOST -H "User-Agent: mcad-controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer MY_KEY" -H "Accept: application/json, /" -H "Content-Type: application/json" 'https://MY_IP_ADDRESS:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions'
I0127 13:52:46.292710 1 round_trippers.go:438] POST https://MY_IP_ADDRESS:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions 403 Forbidden in 1 milliseconds
I0127 13:52:46.292742 1 round_trippers.go:444] Response Headers:
I0127 13:52:46.292751 1 round_trippers.go:447] Content-Type: application/json
I0127 13:52:46.292757 1 round_trippers.go:447] X-Content-Type-Options: nosniff
I0127 13:52:46.292764 1 round_trippers.go:447] Content-Length: 398
I0127 13:52:46.292770 1 round_trippers.go:447] Date: Mon, 27 Jan 2020 13:52:46 GMT
I0127 13:52:46.292794 1 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kube-system:xqueuejob-controller" cannot create customresourcedefinitions.apiextensions.k8s.io at the cluster scope","reason":"Forbidden","details":{"group":"apiextensions.k8s.io","kind":"customresourcedefinitions"},"code":403}

[core] Capability to keep all pods running and restart if any pod fails

Distributed training jobs running large models need all pods/nodes to be up and running during the training, if any pod/node fails then such jobs should be killed, added to the queue, and auto-restarted with desired workers on a different set of nodes.

Appwrappers change directly from Dispatched state to Completed state

We have noticed that at times for the same workload the AW skips few stages and does not have the stages listed below:

  Conditions:
    Last Transition Micro Time:  2022-06-08T21:25:29.441447Z
    Last Update Micro Time:      2022-06-08T21:25:29.441445Z
    Status:                      True
    Type:                        Init
    Last Transition Micro Time:  2022-06-08T21:25:29.441901Z
    Last Update Micro Time:      2022-06-08T21:25:29.441900Z
    Reason:                      AwaitingHeadOfLine
    Status:                      True
    Type:                        Queueing
    Last Transition Micro Time:  2022-06-08T21:25:29.463105Z
    Last Update Micro Time:      2022-06-08T21:25:29.463105Z
    Reason:                      FrontOfQueue.
    Status:                      True
    Type:                        HeadOfLine
    Last Transition Micro Time:  2022-06-08T21:25:40.862757Z
    Last Update Micro Time:      2022-06-08T21:25:40.862755Z
    Reason:                      AppWrapperRunnable
    Status:                      True
    Type:                        Dispatched
    Last Transition Micro Time:  2022-06-08T21:26:50.918204Z
    Last Update Micro Time:      2022-06-08T21:26:50.918202Z
    Reason:                      PodsRunning
    Status:                      True
    Type:                        Running
    Last Transition Micro Time:  2022-06-08T21:27:05.921021Z
    Last Update Micro Time:      2022-06-08T21:27:05.921019Z
    Reason:                      PodsCompleted
    Status:                      True
    Type:                        Completed
  Controllerfirsttimestamp:      2022-06-08T21:25:29.440532Z
  Filterignore:                  true
  Queuejobstate:                 Completed
  Sender:                        before [syncQueueJob] setCompleted
  State:                         Completed

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.