Retry/Middlewares + CQRS busted? about watermill

[m110]

Retry middleware is meant for temporary issues, like some network error while saving to database or an error from an external service. In most cases the Pub/Sub itself will re-deliver the message as it won’t be acked, but using middleware gives you a bit more control.

PRs are welcome, of course! 🙂 I’d consider making this a separate middleware, something like DiscardOnError. Possibly could also allow users to take some action before, like log the message. Although in most cases you’d probably want to use PoisonQueue anyway, as it might be hard to distinguish between temporary retryable errors and non-fixable errors.

Also, there’s a common pattern of using both of them in a chain: first, you try to retry the message for several times, and if that doesn’t work that probably means the issue is with message format or other serious error. So PoisonQueue catches it and stores on a separate topic. This is as simple, as adding both middlewares one after another (PoisonQueue first).

[m110]

Hey @smazurov.

The exit from Retry middleware, when the handler returns no error is this line:

watermill/message/router/middleware/retry.go

Line 71 in 8840d3f

producedMessages, err = h(msg)

The line you mentioned will return the error after several retries, when break happens out of the loop.

The change you proposed would make the middleware return no error after several attempts of retry. This would make you lose the message (because it would be acked). If that's the behaviour you're looking for, take a look at PoisonQueue middleware: https://watermill.io/docs/middlewares/#poison

Does that make sense?

[smazurov]

That makes sense, for my own education, whats the point of retry middleware? I guess the assumption is that one handler could be faulty while another handler can handle that message? Ideally what I'm looking for is errors that I know won't ever be resolved for that message (say extra validation), so I want to just get rid of it. PoisonQueue should work, so thanks for that!

Alternatively, would you be open to accepting a PR that adds an option to drop failed messages?

[roblaszczak]

Hello @smazurov, is everything clear now?
I'm closing the issue but if it shouldn't please let me know :)