thebrunolopes / condu Goto Github PK

I'm creating an issue for this change in condu because it might change a vast proportion of code.
The problem is:

• When a worker process (child process that polls tasks from conductor server) is killed by the OS, for example due to OOM, there is no way to update the task being treated by the process that is gone as "FAILED".
• The process that is killed is set as a Zombie process and the task type that was being handled by this process is never polled again from this worker.

The solution that I will provide is:

• When child processes are triggered, they will be triggered from different threads that maintain a pipe open to the process that is triggered.
• Whenever the process receives a task from Conductor server it communicates this task with the thread that triggered the process.
• If the child process is killed, for example due to OOM, the thread that created the process can observe this and can update the task as FAILED and create a new process to replace the one killed before.

For any doubts or suggestions, comment please.

Currently, when condu pulls a task to be processed in a worker, it is not ACKnowledging the task. Thus if the task takes more than responseTimeoutSeconds, it will be scheduled again by conductor-server after responseTimeoutSeconds and can be pulled by another worker even if the first worker that pulled the task is still working on it.

I will fix this issue by ACKnowledging the task after pollForTask if the task is not None.

More information about ACKnowleging tasks here: https://netflix.github.io/conductor/runtime/