Workers quit before all pending tasks are done about luigi

[erikbern]

Yeah this is a feature/bug depending on how you look at it. We thought about doing some kind of sleep/retry or even long-polling. However we thought it was easier to just trigger the job repeatedly until it builds. Typically we run all our workflows from cron every minute. If the same workflow is already running, it will just exit. If nothing is running, it will traverse the dependency graph again and try to schedule everything.

Sleep/retry should be fairly easy to implement though, but I think it should be configurable. There's a risk you might create a lot of spinning processes. There are also some corner cases like you could end up having 100 processes waiting for Block to finish so they can build Dep, but then once Block finishes, only one process will end up building Dep and the rest will exit.

[jimjh]

Oops forgot to close this. Added a flag to work around this in #225.