problem of pytorch operator about training-operator

[johnugeorge]

Yes. It is based on HPA. You can use scale metric as well for HPA to pick up automatically.

eg:
https://github.com/kubeflow/training-operator/blob/master/examples/pytorch/elastic/imagenet/imagenet.yaml

[panowo]

@erictanjn The problem in GPU is that the resource metrics are CPU related which do not capture the GPU utilization, how can we adjust replicas then ?

Some custom GPU metrics can be used for HPA, such as DCGM

[erictanjn]

hi,
I noticed HPA use crd's spec.replicas as currentReplicas but there is not .Spec.Replicas in pytorchjob crd. So I'm a little bit confused how it works. How HPA controller get the currentReplicas of a pytorchjob?

the code in HPA controller as following:
currentReplicas := scale.Spec.Replicas

[tenzen-y]

The HPA controller will obtain the PyTorchJob replicas from .spec.pytorchReplicaSpecs.Worker.replicas:

training-operator/manifests/base/crds/kubeflow.org_pytorchjobs.yaml

Line 8405 in 11b7a11

specReplicasPath: .spec.pytorchReplicaSpecs.Worker.replicas

[kuizhiqing]

Hi @erictanjn , can you share some detail about your user case in detail plz, AFAK, HPA is more like CPU oriented, do you work in GPU case and how you expect it works in production ?

Thanks in advance.

[erictanjn]

The HPA controller will obtain the PyTorchJob replicas from .spec.pytorchReplicaSpecs.Worker.replicas:

training-operator/manifests/base/crds/kubeflow.org_pytorchjobs.yaml

Line 8405 in 11b7a11

specReplicasPath: .spec.pytorchReplicaSpecs.Worker.replicas

do I need to do some work in HPA controller? Or it will obtain the PyTorchJob replicas from .spec.pytorchReplicaSpecs.Worker.replicas automatically？

[erictanjn]

Hi @erictanjn , can you share some detail about your user case in detail plz, AFAK, HPA is more like CPU oriented, do you work in GPU case and how you expect it works in production ?

Thanks in advance.

I use it in GPU case. So I am a little bit worried it cannot get the PyTorchJob replicas automatically.

[kuizhiqing]

@erictanjn The problem in GPU is that the resource metrics are CPU related which do not capture the GPU utilization, how can we adjust replicas then ?

[tenzen-y]

The HPA controller will obtain the PyTorchJob replicas from .spec.pytorchReplicaSpecs.Worker.replicas:

training-operator/manifests/base/crds/kubeflow.org_pytorchjobs.yaml

Line 8405 in 11b7a11

specReplicasPath: .spec.pytorchReplicaSpecs.Worker.replicas

do I need to do some work in HPA controller? Or it will obtain the PyTorchJob replicas from .spec.pytorchReplicaSpecs.Worker.replicas automatically？

It's automatically.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.