Comments (3)
Hey @ShuyangCao. Nicely done! Keep up your good works using ts!
In fact, logging is handled by the client, not the server, so you can still see your progress via offline log files in /tmp
. -t/-c
simply read from these files. You can sort the files to find the wanted log. When the server crashes, all information about the jobs is gone, so it's impossible to use -t/-c [jobid]
anymore.
Above all, the crash should not happen at all. This is probably due to the poor error handling of GPU query. I pushed a simple fix for this in the branch gpu-err
. Please check and see if the error can be handled or not.
Btw, there's a log file in /tmp
which has the format socket-ts.<uid>.error
. Could you please let me know the error message that the server gave when the query was unsuccessful?
from task-spooler.
Thanks! Yes, I can still check the files in /tmp
, but I am not sure which job each file corresponds to.
The error messages are:
-------------------Error
Msg: Error calling recv_msg in c_check_version
errno 104, "Connection reset by peer"
date Tue Dec 20 12:38:25 2022
pid 82645
type CLIENT
-------------------Error
Msg: Failed to get GPU handle for GPU 3: GPU is lost
errno 2, "No such file or directory"
date Wed Dec 21 07:04:18 2022
pid 4351
type SERVER
from task-spooler.
Thanks for the reply @ShuyangCao. The first error probably is caused by a message from an orphan client sent to a restarted server. This is harmless in most cases. The second error will return a NULL
pointer, which causes the server crash as you experienced. The patch I pushed should be able to handle this error.
Please let me know if there's still any problem.
from task-spooler.
Related Issues (20)
- install breaks without CUDA HOT 1
- Bug: cannot add a very long command to queue HOT 14
- Structured output HOT 2
- Please edit the README HOT 2
- make cpu giving error: implicitly declaring library function 'snprintf' with type 'int HOT 2
- Evaluate $(...) in commands at run not at enqueue HOT 2
- Using `-n` `-f` flags: pass through SIGINT (and other signals?) HOT 1
- ts -F stochastically crashes the server HOT 3
- Contributors HOT 7
- GUI addon link point to 404 HOT 3
- asynchronous launch HOT 6
- Timeout HOT 3
- json format for listing jobs HOT 1
- Unable to redirect output from command line HOT 2
- Enhancement request: support for priorities HOT 2
- Enhancement request: ability to postpone jobs HOT 2
- Enhancement request: bigger queue size HOT 9
- GPU selection scheme
- Copyright HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from task-spooler.