PytorchJob restartPolicy: ExitCode does not honor backoffLimit for retryable errors about training-operator

[kellyaa]

On reviewing this again, there's some possible solutions I can think of:

1. On this code that checks if backofff limit is exceeded, have it look for job restart events instead of container restarts. In looking at the code though, I don't see that this information about the job's events is currently available in this context.
2. Forget this effort until the incorporation of batchv1 jobs is complete. That way, we can rely on the Job's restart policy (similar to how OnFailure is relying on the pod's own restart policy) instead of inventing a way propreitary to the training operator.

My understanding is the date for (2) is unknown and is pending job success policy becoming beta in Kubernetes (no date). Selfishly, I'd like to use this feature sooner than later, so if there's any other short term fix that would be great. What do you think about 1). ?

@andreyvelich @tenzen-y

[tenzen-y]

On reviewing this again, there's some possible solutions I can think of:

1. On this code that checks if backofff limit is exceeded, have it look for job restart events instead of container restarts. In looking at the code though, I don't see that this information about the job's events is currently available in this context.
2. Forget this effort until the incorporation of batchv1 jobs is complete. That way, we can rely on the Job's restart policy (similar to how OnFailure is relying on the pod's own restart policy) instead of inventing a way propreitary to the training operator.

My understanding is the date for (2) is unknown and is pending job success policy becoming beta in Kubernetes (no date). Selfishly, I'd like to use this feature sooner than later, so if there's any other short term fix that would be great. What do you think about 1). ?

@andreyvelich @tenzen-y

This is actually a specification, not a bug. You can see a similar feature in the batch/v1 Job w/ PodFailurePolicy action=Ignore.
So, I think that this is a feature request similar to the batch/v1 Job w/ PodFailurePolicy action=Count.

[tenzen-y]

Since selecting (1) means breaking change, we can not select the approach.
So, adding a new field like the batch/v1 Job action field might be better.

[tenzen-y]

/kind feature

[tedhtchang]

Hi @tenzen-y Are you going to work on this API change ? or how far have you started so far.