[KED-974] Running the pipeline from the last failed node about kedro

[lorenabalan]

Thanks for raising this @anuarora1990, we'll look into finding a suitable solution! I've updated the title with our internal ticket number to make it easier to track.

[lorenabalan]

Sorry @anuarora1990 this should've been closed a while ago. Starting with Kedro version 0.15.1 there is a resume command that's suggested in the logs after a pipeline fails (kedro run --from-inputs a,b,c). It's important to note that the command won't work if the inputs are MemoryDatasets. We might revisit this in the future, but for now I will be closing this issue, please feel free to open a new one if you have additional feedback.

[Minyus]

@anuarora1990

I suffered from the same issue and discussed at:

#30
#60

To solve the pain point, I implemented pipeline resuming option as a wrapper of Kedro.
Here is the GitHub repository:

https://github.com/Minyus/pipelinex

You do not need to specify the Kedro pipeline nodes using from-inputs option if you set only_missing option to True.

Hope this helps.