Stopping a pipeline on error about broadway

[josevalim]

Broadway was done with the assumption of independent events. Even the idea of partitions is there to guarantee ordering but not to guarantee logical grouping in case of failures. So “stopping a partition” doesn’t make much sense, as you would also stop unrelated events. But I also think what you want can be achieved with a custom producer. The trick is for the producer to hold events for a given partition until the previous element was successfully processed and so on. I believe this is what Kafka consumers do.

…

-- *José Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D

[slashdotdash]

Thanks for the feedback @josevalim I’ll look into using a custom producer. Initially it will be simpler just to stop all message processing on error (if requested).

[josevalim]

Perfect, if you have any follow up questions, feel free to post them here! Btw, which source will you be consuming data from?

PS: I am closing it though as there is nothing to do in Broadway, but we can continue the convo.

[slashdotdash]

The messages will be events from Event Store subscriptions (Postgres database).