Separate logging and queueing? about task-spooler

[justanhduc]

Hey @ShuyangCao. Nicely done! Keep up your good works using ts!

In fact, logging is handled by the client, not the server, so you can still see your progress via offline log files in /tmp. -t/-c simply read from these files. You can sort the files to find the wanted log. When the server crashes, all information about the jobs is gone, so it's impossible to use -t/-c [jobid] anymore.

Above all, the crash should not happen at all. This is probably due to the poor error handling of GPU query. I pushed a simple fix for this in the branch gpu-err. Please check and see if the error can be handled or not.

Btw, there's a log file in /tmp which has the format socket-ts.<uid>.error. Could you please let me know the error message that the server gave when the query was unsuccessful?

[ShuyangCao]

Thanks! Yes, I can still check the files in /tmp, but I am not sure which job each file corresponds to.

The error messages are:

-------------------Error
 Msg: Error calling recv_msg in c_check_version
 errno 104, "Connection reset by peer"
date Tue Dec 20 12:38:25 2022
pid 82645
type CLIENT
-------------------Error
 Msg: Failed to get GPU handle for GPU 3: GPU is lost
 errno 2, "No such file or directory"
date Wed Dec 21 07:04:18 2022
pid 4351
type SERVER

[justanhduc]

Thanks for the reply @ShuyangCao. The first error probably is caused by a message from an orphan client sent to a restarted server. This is harmless in most cases. The second error will return a NULL pointer, which causes the server crash as you experienced. The patch I pushed should be able to handle this error.

Please let me know if there's still any problem.