[SDK] Get Job Pods Events about training-operator

[kuizhiqing]

what would be easier to understand with_events or verbose flag ?

@andreyvelich Well, you'er right, it would be better to use verbose option without exposing event definition.

[terrytangyuan]

Good idea. +1

[tenzen-y]

It sounds good!

Also, at the same time, showing the event of FrameworkJob (e.g., TFJob) at the top might be helpful.
@andreyvelich WDYT?

[johnugeorge]

Should this be a different API for providing more clarity ?

[kuizhiqing]

It would be helpful!

Maybe we should add new API get_job_events to print events of job and pods. And we add new arg with_events=False in API get_job_logs to make it possible to get all the information in the same API.

[andreyvelich]

Also, at the same time, showing the event of FrameworkJob (e.g., TFJob) at the top might be helpful.

@tenzen-y That sounds good. The question is how to identify which Job user created ? get_job_logs doesn't have job_type as an input argument, and we just check which pod has these labels:

training.kubeflow.org/job-name=my-job
training.kubeflow.org/job-role=master

The same labels could have multiply jobs (e.g. PyTorchJob, XGBoostJob).

That ties to my other question, if we are going to introduce mandatory job_type argument to our get_job_logs API, should we follow the same pattern for all APIs ?
E.g. instead of get_pytorchjob, get_tfjob, we are going to have a single API called: get_job which takes job_type as a mandatory argument and we are going to get appropriate job based on this type.

We can do the same for all other APIs: create_job, create_job_from_func, get_job, delete_job`, etc.

After refactoring our SDK: #1719, I noticed that it is very confusing for the user that we have some CRUD operations job specific (e.g. create_tfjob), but some of them are not (e.g. get_job_pod_names, get_job_logs).
I can create separate issue to discuss this, and I am going to provide more feedback from the users soon.
cc @kubeflow/wg-training-leads

[andreyvelich]

Should this be a different API for providing more clarity ?

I am not sure, if users who are not familiar with Kubernetes should know differences between events and logs.
Usually, when Data Scientists create a ML Job, they want to directly check the logs from this job (e.g. run get_job_logs API).
Otherwise, we should somehow explain them if get_job_logs API fails, they should run get_job_events API.
WDYT @johnugeorge ?

And we add new arg with_events=False in API get_job_logs to make it possible to get all the information in the same API.

I like the idea @kuizhiqing, what would be easier to understand with_events or verbose flag ?

[johnugeorge]

+1 Agree with you @andreyvelich

[tenzen-y]

The same labels could have multiply jobs (e.g. PyTorchJob, XGBoostJob).

That ties to my other question, if we are going to introduce mandatory job_type argument to our get_job_logs API, should we follow the same pattern for all APIs ?
E.g. instead of get_pytorchjob, get_tfjob, we are going to have a single API called: get_job which takes job_type as a mandatory argument and we are going to get appropriate job based on this type.

We can do the same for all other APIs: create_job, create_job_from_func, get_job, delete_job`, etc.

After refactoring our SDK: #1719, I noticed that it is very confusing for the user that we have some CRUD operations job specific (e.g. create_tfjob), but some of them are not (e.g. get_job_pod_names, get_job_logs).
I can create separate issue to discuss this, and I am going to provide more feedback from the users soon.

@andreyvelich Thanks for the clarification.
I agree with you. Let's work on events for XXXJob in another issue.

[andreyvelich]

/assign @andreyvelich

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/assign @andreyvelich