Pipes should fail after an exception about pypette

[csurfer]

@kstrauser: I have a different view to present here. I don't think it should fail. The reason I think so is one of the jobs might be a cleanup job which is quite needed in a lot of places. If the failure of a stage compels the next stage to fail, we may end up in a situation where the cleanup jobs are not executed.

Having said that I believe we should have control over dependency and choice. So, I have been thinking of Job having an optional parameter called gate. This gate is a callable of some sort which returns a boolean which would decide whether the job needs to be executed or not.

j1 = Job(job1)
# Lets assume job1 pushes its state into a queue or some variable x.
def gate_method():
    flag = True
    # flag value is set to variable x or popped value from queue
    return flag
j2 = Job(job2, gate=gate_method)

When job 2 is being run it will check for value returned by gate_method and only if it is true will execute job2.

That will introduce the notion of dynamic decision in pipeline. What do you think?

[kstrauser]

That’s a really good point about cleanup jobs! I think the gate idea sounds wonderful, though. What if there was also a function like last_job_succeeded() that returns true if and only if the previous job completed without raising an exception. Perhaps that could even be the default value, like gate=last_job_succeeded, so that by default a failure stop execution but that it would be trivially easy to turn it off for cleanup jobs?

[csurfer]

@kstrauser : Have added a bunch of new stuff including exception handling modes and dependency. Look at examples/ for the usage. Still testing a couple of things after which will release officially. Give it a whirl and let me know.

[csurfer]

v0.0.9 should resolve this issue.

[kstrauser]

I'll take a look. Thanks for following up!